/r/datacleaning

Photograph via snooOG

Data scientists can spend up to 80 percent of their time correcting data errors before extracting value from the data.

We at /r/datacleaning are interested in data cleaning as a preprocessing step to data mining. This subreddit is focused on advances in data cleaning research, data cleaning algorithms, and data cleaning tools. Related topics that we are interested in include: databases, statistics, machine learning, data mining, AI, visualization, etc.

Garbage in, garbage out! Data scientists can spend up to 80 percent of their time correcting data errors before extracting value from the data.

We at /r/datacleaning are interested in data cleaning as a preprocessing step to data mining. This subreddit is focused on advances in data cleaning research, data cleaning algorithms, and data cleaning tools. Related topics that we are interested in include: databases, statistics, machine learning, data mining, AI, information theory, information retrieval, pattern recognition, NLP, data visualization, etc.

Related subreddits :

/r/machinelearning /r/etl

/r/datacleaning

4,530 Subscribers

2

The Future of Image Annotation Emerging Trends and Innovations for Businesses

0 Comments
2024/04/09
10:50 UTC

0

What does it imply when the total cost is negative, the unit selling price is positive and the order is 0? I am trying to clean data in Excel.

ORDER QUANTITY | UNIT SELLING PRICE| TOTAL COST

0 | 151.47 | -86.9076

0 | 690.89 | -1002.1401

0 | 822.75 | -978.8337

I am trying to clean a dataset and wanted to understand if it makes sense or if I should delete it from the table. There are about 28% of total entries with such data. It won't make sense to delete 28% either. Please drop your suggestions and understanding.

0 Comments
2024/04/06
18:04 UTC

3

Strategies for Improving Data Quality Through Data Cleansing Services

0 Comments
2024/04/05
15:32 UTC

1

Pricing Inquiry for Data Cleaning and Analysis Service with Databricks and PySpark Expertise

Hello,

I'm currently exploring options for professional data cleaning and analysis services, particularly those utilizing Databricks and PySpark expertise. I have a dataset that requires thorough cleaning to address inconsistencies and erroneous data, followed by in-depth analysis to extract valuable insights for my business.

Here's a breakdown of the tasks I'm looking to outsource:

  1. Initial Evaluation: Assessing my dataset to identify data quality issues.
  2. Data Cleaning: Applying advanced data cleaning techniques to rectify inconsistencies and erroneous data.
  3. Databricks Analysis: Utilizing Databricks for large-scale data analysis, optimizing processing performance.
  4. PySpark Development: Writing PySpark scripts for efficient processing and analysis of distributed data.
  5. Reporting and Insights: Generating detailed reports and providing insights based on the analysis performed.
  6. Continuous Optimization: Recommending strategies for ongoing improvement of data quality and analysis processes.

I understand that the cost of such services can vary depending on factors such as the complexity of the dataset, the volume of data, and the specific requirements of the analysis. However, I would appreciate any ballpark estimates or insights from forum members who have experience with similar projects.

Additionally, if you have recommendations for reputable service providers or consultants specializing in data cleaning and analysis with Databricks and PySpark, please feel free to share them.

Thank you in advance for your assistance!

0 Comments
2024/03/23
03:26 UTC

1

Expert Data Cleaning and Formatting Services by Damco Solutions

Greetings, fellow data enthusiasts!

Are you tired of grappling with messy, unstructured data? Look no further! Damco Solutions offers top-notch Data Cleaning and Formatting Services to streamline your data management processes.

With our years of expertise in data science and analytics, we understand the importance of clean, accurate data for making informed business decisions. Whether you're dealing with outdated records, inconsistent formatting, or incomplete datasets, our team of skilled professionals is here to help.

Why choose Damco Solutions for your data cleaning needs?

1. Precision and Accuracy: We employ advanced algorithms and manual verification techniques to ensure that your data is cleaned and formatted with the utmost precision.

2. Customized Solutions: Every dataset is unique, and we tailor our services to meet your specific requirements. Whether you need data deduplication, standardization, or validation, we've got you covered.

3. Data Security: We prioritize the security and confidentiality of your data. Rest assured that your sensitive information is safe in our hands.

4. Timely Delivery: We understand the importance of deadlines. Our efficient workflow ensures timely delivery without compromising on quality.

5. Cost-Effective: Our services are competitively priced, offering excellent value for your investment.

Ready to experience the difference that clean, well-formatted data can make in your business operations? Visit our service page to learn more about our offerings.

Don't let messy data hold you back. Trust Damco Solutions to unlock the full potential of your data assets!

Cheers,

0 Comments
2024/03/12
11:15 UTC

1

Cleaning header/footer text from OCR data

Hello! I have a collection of OCR text from about a million journal articles and would appreciate any input on how I can best clean it.

First, a bit about the format of the data: each article is stored as an array of strings where each string is the OCR output for each page of the article. The goal is to have a single large string for each article, but before concatenating the strings in these arrays, some cleaning needs to be done at the start and end of each string. Because we're talking about raw OCR output, and many journals have things like journal titles, page numbers, article titles, author names, etc. at the top and/or bottom of each page, and those have to be removed first.

The real problem, however, is that there is just so much variation in how journals do this. For example, some alternate between journal title and article tile at the top of each page with page numbers at the bottom, some alternate between page numbers being at the top and the bottom of each page, and the list goes on. (So far, I've identified 10 different patterns just from examining 20 arrays.) This is further complicated by most articles having different first and sometimes last pages, tables and captions, etc.

At this point, I could keep going to identify patterns, write some regex to detect what pattern is present, then clean accordingly. But I also wonder if there's a more general approach, like searching for some kind of regularity, either across pages or (more commonly) every other page, but I'm not quite sure how I should approach this task.

Any suggestions would be greatly appreciated!

1 Comment
2024/03/07
22:24 UTC

5

Looking to create a "Clean Data" definition

Hi,

Just wondering what requirements or checklist items people would suggest for a definition of Clean Data ready to be used in machine learning? Akin to "tidy data", but for modelling. I.e.

  • There should be no string fields. All data should be either in a numeric form, or as a categorical data type

etc

I know this will likely be opinionated, hence wanting to "crowd source" it 😃

Feel free to disagree with any statements, as I imagine there will be differences

0 Comments
2024/02/29
12:39 UTC

2

Most appropriate tool for data cleaning research dataset

Hi everyone!

I am cleaning a dataset from a cross-sectional survey. It has 1100 columns and 600 rows.

Currently, I am using excel and manually cleaning everything (converting texts to numerical codes, individually checking the columns if there are wrong encodings using the filter function etc.) then documenting the changes one by one on a google document. I'm also building a data dictionary as I go.

I was wondering if anyone can recommend a better way to do this. I want to learn the good/best data cleaning practices.

Thank you very much for the help!

3 Comments
2024/02/16
23:52 UTC

0

Beyond Scrubbing The Competitive Advantages of Data Cleansing Services

0 Comments
2023/12/29
11:12 UTC

1

Data Cleaning Freelancer

Hey everyone,

I'm a sophomore studying data science and I've been digging into ways to earn money online. I stumbled upon the idea of freelancing my data cleaning skills, and it seems like an exciting avenue. Though I'm still learning, I'm a quick learner and confident that I can get proficient in data cleaning soon.

I'm keen to get hands-on experience and was wondering if anyone would be open to taking me under their wing as an apprentice or offering advice on where to begin.

While I'm still early in my studies, I've worked on a few exploratory data analyses for my classes. These involved cleaning data and using RStudio to create graphs.

I'm eager to turn this interest into a reality. Any guidance or tips on how to kickstart a career in freelancing data cleaning would be hugely appreciated!

Thanks in advance for any help or advice you can offer!

1 Comment
2023/12/23
22:23 UTC

2

AI tool to extract product characteristics

Hello everyone,

I am trying to clean up some data from our ERP systems regarding our items. I am working for a furniture company, we do have different characteristics that compose a product (size/timber/fabric and so on). So far, those characteristics has been input all in one description field. I'd like to extract those information and assign it to the new correct field (one field per characteristic). Maybe some AI tools might be able to help in that process? I am not a developer / technical IT.

2 Comments
2023/12/05
03:11 UTC

1

Success Story: How data cleaning tools helped support my project

Disclaimer: This is a personal project I did, made possible with RPA (UiPath web scraping). The stats come from SA Rugby website & I developed automation flows to get the stats, player bio & profile pictures from the same website. I used PowerQuery to transform the output & to debug issues & finally Tableau for visualisation. I highly recommend getting comfortable with Power Query, you can do so much with it!

Hi everyone, I'd like to share a personal project I did about the Springboks RWC Campaign. I'd love to get your feedback as PowerBI people, to get your unique perspective. We only use Tableau at so I thought I'd overcome confirmation bias by getting your guys' opinions.

The project is basically match stats for all the games the Springboks played in all championships in 2023. You can see those who are consistently performing well. The stats come from SA Rugby

Each match has highlight reels of the players' game contributions (71 total). The project also covers all the matches that the Boks under Rassie have played NZ (5 Wins, 5 Losses & 1 Draw).

Ultimately, the project shows how tough this World Cup was & the pressure the team faced, especially in the knockout phases.

PS. I think this would be great for those new to rugby, since it covers the biggest matches in the sport with highlight reels to see the entertaining stuff.

You can check out the full work here: https://public.tableau.com/views/Springboks2023RugbyWorldCupCampaign/TheSpringboks2023Campaign?:language=en-US&:display_count=n&:origin=viz_share_link

Final vs NZ

Semi Final vs England

Quarter Final vs France

0 Comments
2023/11/06
17:19 UTC

1

"Cleaning Call Center Data: Seeking Guidance/Help"

Hello everyone,

I am currently working on a call center trend dashboard project, and I've encountered an issue with multiple blank cells in the data. I'm unsure about the best approach to handle this. Should I delete rows with multiple blank cells, or should I use statistics to fill these blank cells?

I would greatly appreciate your guidance and suggestions on this matter. Your assistance would be invaluable. Thank you in advance!

Project Task :

Create a dashboard in Power BI for Claire that reflects all relevant Key Performance Indicators (KPIs) and metrics in the dataset

Possible KPIs include (to get you started, but not limited to):

  • Overall customer satisfaction
  • Overall calls answered/abandoned
  • Calls by time
  • Average speed of answer
  • Agent’s performance quadrant -> average handle time (talk duration) vs calls answered

Some info about data:

Total rows-5000

Total column :10

snapshot of data

"Total rows having missing values: 946 Each of the 946 rows has 3 blank/missing cells.

Please guide me on the approach I should take to clean this data.

Note: The blank column is just a temporary column used to check how many cells are blank in each row."

TL;DR:Seeking advice on handling data with many missing values (946 rows, 3 blank cells each) for a call center trend dashboard project. Also, tasked with creating a Power BI dashboard for Claire, highlighting KPIs and metrics. Please assist. Thanks!

2 Comments
2023/11/06
07:52 UTC

1

How to handle missing categorical values with more than 5% missing data?

I am upskilling in the field of data science. Recently started practicing on Kaggle datasets. Picked up a dataset which have more categorical columns than numerical and these columns have more that 5% (upto 60% null values in some columns) null values. I am confused about what technique to use on them. Cannot find resources where handling object columns specifically is focused upon. Any help please? can anyone suggest a book or website or just tell me how to proceed with this?

4 Comments
2023/07/12
19:43 UTC

7

How to get started with python for data analysis?

If you're embarking on the odyssey of studying Python data analysis, commence by acquiring a mastery of the rudiments of Python programming.

Once you've attained a level of proficiency with Python, plunge into the depths of indispensable libraries such as NumPy for numerical computation and Pandas for data manipulation. Engage in practical exercises utilizing authentic datasets to accrue experiential knowledge, and refine your prowess in data visualization employing Matplotlib and Seaborn.

Delve into the realm of statistical analysis using the comprehensive tools provided by SciPy, and contemplate augmenting your skill set with other pertinent libraries such as scikit-learn for machine learning. Engross yourself in online communities, undertake ambitious projects, and perpetually pursue learning and diligent practice to ascend to a zenith of expertise in Python data analysis—a gratifying pursuit that unveils the portals to unearthing invaluable insights from data.

To get you started, I will highly recommend you look at these articles.

Exploratory Data Analysis and visualization practical example:

https://link.medium.com/FYuBpTyvCAb

Data cleaning with python (a practical example)

https://link.medium.com/GBsdtEFvCAb

How to make data Visualization in python

https://link.medium.com/6rWH2nKvCAb

Python data cleaning made easy

https://link.medium.com/6rWH2nKvCAb

Sales Statistical analysis with python

https://link.medium.com/ZGx7NDRvCAb

https://link.medium.com/OidaOBUvCAb

Python Web App Development: Unleashing the Power of Simplicity and Flexibility

https://medium.com/@mondoa/python-web-app-development-unleashing-the-power-of-simplicity-and-flexibility-d34e9d1bd658

Enhancing Your Web Application with Python’s Data Analysis Tools

https://medium.com/@mondoa/enhancing-your-web-application-with-pythons-data-analysis-tools-2ecd0af29027

The Ultimate Python 3 Guide: Everything You Need to Know

https://medium.com/@mondoa/enhancing-a-comprehensive-python-3-tutorial-b8102f0cfcc4

0 Comments
2023/07/01
18:34 UTC

1

Help Extracting Data from XML

I need help with figuring out the best tool to do so extraction of data. I work on a Wiki and I am able to download XMLs of large sets of pages. For this to be any use to us, I need to be able to put them in Excel to turn them into CSV files to be able to reupload them after I've fixed or added more data. Here's an example of what I can manually do right now to turn it into a format I need for the CSV file:

First I download the XML File. This example only has 3 pages in it, but usually there are hundreds. It looks something like this:

mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
<siteinfo>
<sitename>FamilySearch Wiki</sitename>
<dbname>wiki_en</dbname>

<base>https://www.familysearch.org/en/wiki/Main\_Page</base> <generator>MediaWiki 1.35.8</generator> <case>first-letter</case> <namespaces> <namespace key="-2" case="first-letter">Media</namespace> <namespace key="-1" case="first-letter">Special</namespace> <namespace key="0" case="first-letter" /> <namespace key="1" case="first-letter">Talk</namespace> <namespace key="2" case="first-letter">User</namespace> <namespace key="3" case="first-letter">User talk</namespace> <namespace key="4" case="first-letter">FamilySearch Wiki</namespace> <namespace key="5" case="first-letter">FamilySearch Wiki talk</namespace> <namespace key="6" case="first-letter">File</namespace> <namespace key="7" case="first-letter">File talk</namespace> <namespace key="8" case="first-letter">MediaWiki</namespace> <namespace key="9" case="first-letter">MediaWiki talk</namespace> <namespace key="10" case="first-letter">Template</namespace> <namespace key="11" case="first-letter">Template talk</namespace> <namespace key="12" case="first-letter">Help</namespace> <namespace key="13" case="first-letter">Help talk</namespace> <namespace key="14" case="first-letter">Category</namespace> <namespace key="15" case="first-letter">Category talk</namespace> <namespace key="102" case="first-letter">Property</namespace> <namespace key="103" case="first-letter">Property talk</namespace> <namespace key="106" case="first-letter">Form</namespace> <namespace key="107" case="first-letter">Form talk</namespace> <namespace key="108" case="first-letter">Concept</namespace> <namespace key="109" case="first-letter">Concept talk</namespace> <namespace key="112" case="first-letter">smw/schema</namespace> <namespace key="113" case="first-letter">smw/schema talk</namespace> <namespace key="114" case="first-letter">Rule</namespace> <namespace key="115" case="first-letter">Rule talk</namespace> <namespace key="200" case="first-letter">Policy</namespace> <namespace key="201" case="first-letter">Policy Talk</namespace> <namespace key="202" case="first-letter">Shared Category</namespace> <namespace key="203" case="first-letter">Shared Category Talk</namespace> <namespace key="274" case="first-letter">Widget</namespace> <namespace key="275" case="first-letter">Widget talk</namespace> <namespace key="420" case="first-letter">GeoJson</namespace> <namespace key="421" case="first-letter">GeoJson talk</namespace> <namespace key="460" case="case-sensitive">Campaign</namespace> <namespace key="461" case="case-sensitive">Campaign talk</namespace> <namespace key="828" case="first-letter">Module</namespace> <namespace key="829" case="first-letter">Module talk</namespace> <namespace key="2300" case="first-letter">Gadget</namespace> <namespace key="2301" case="first-letter">Gadget talk</namespace> <namespace key="2302" case="case-sensitive">Gadget definition</namespace> <namespace key="2303" case="case-sensitive">Gadget definition talk</namespace> <namespace key="3100" case="first-letter">GuidedResearch</namespace> <namespace key="3101" case="first-letter">GuidedResearch Talk</namespace> <namespace key="3102" case="first-letter">AFOG</namespace> <namespace key="3103" case="first-letter">AFOG Talk</namespace> <namespace key="3104" case="first-letter">Indonesia</namespace> <namespace key="3105" case="first-letter">Indonesia Talk</namespace> <namespace key="3106" case="first-letter">Mongolian</namespace> <namespace key="3107" case="first-letter">Mongolian Talk</namespace> <namespace key="3108" case="first-letter">Norwegian</namespace> <namespace key="3109" case="first-letter">Norwegian Talk</namespace> <namespace key="3110" case="first-letter">GR</namespace> <namespace key="3111" case="first-letter">GR Talk</namespace> </namespaces> </siteinfo> <page> <title>Cosalá, Sinaloa, Mexico Genealogy</title> <ns>0</ns> <id>386351</id> <revision> <id>5345468</id> <parentid>5345405</parentid> <timestamp>2023-05-31T19:28:51Z</timestamp> <contributor> <username>Amberannelarsen</username> <id>490153</id> </contributor> <minor/> <comment>Text replacement - "\&amp;#243;" to "ó"</comment> <origin>5345468</origin> <model>wikitext</model> <format>text/x-wiki</format> <text bytes="1890" sha1="dnzksnhh9qwij21kx164z04wnsci09g" xml:space="preserve">{{breadcrumb | link1=\[\[Mexico Genealogy|Mexico\]\] | link2=\[\[Sinaloa, Mexico Genealogy|Sinaloa\]\] | link3= | link4= | link5=\[\[Cosalá, Sinaloa, Mexico Genealogy|Cosalá\]\] }} Guide to '''Municipality of Cosalá family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records. ==History== \*El territorio donde actualmente se ubica Cosalá, estuvo ocupado por pueblos prehispánicos que se asentaron principalmente en la rivera de los ríos, como lo fueron los grupos indígenas Tepehuanes, Acaxees y Xiximes. \*El municipio de Cosalá fue fundado el 13 March 1562. \*El municipio de Cosalá tiene una población de aproximadamente 17.000 personas.\&lt;ref\&gt;Wikipedia contributors, “Municipio de Cosalá” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Cosal%C3%A1. accessed 25 February2021.\&lt;/ref\&gt; ==Localities within Cosalá== {| style="width:100%; vertical-align:top;" |- | \&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"\&gt; \&lt;li\&gt;Cosalá\&lt;/li\&gt; \&lt;li\&gt;El Rodeo\&lt;/li\&gt; \&lt;li\&gt;La Llama\&lt;/li\&gt; \&lt;/ul\&gt; |} ==Civil Registration== \*'''1867-1929''' {{FHL|2819510|title-id|disp=Mexico, Sinaloa, Cosalá, Civil Registration, 1867-1929}}(\*) at FamilySearch Catalog — images ==Parish Records== \*'''1777-1966''' {{FHL|263768|title-id|disp=Iglesia Católica. Santa Ursula (Cosala, Sinaloa) Parish Records, 1777-1966}}(\*) at FamilySearch Catalog — images \*'''1874-1920''' {{FHL|260349|title-id|disp=Iglesia Católica. Santa Ursula (Cosalá, Sinaloa) Parish Records, 1874-1920}}(\*) at FamilySearch Catalog — images ==Census== ==Cemeteries== \*Cementerio de San Juan Cosala :\*\[https://www.findagrave.com/cemetery-browse/Mexico/Sinaloa/Cosal%C3%A1-Municipality?id=county\_13453 Find a Grave\] ==References== \&lt;references/\&gt; \&lt;br\&gt;\&lt;br\&gt; \[\[es:Cosalá, Sinaloa, Mexico Genealogy\]\] \[\[Category:Sinaloa, Mexico\]\]</text> <sha1>dnzksnhh9qwij21kx164z04wnsci09g</sha1> </revision> </page> <page> <title>Mocorito, Sinaloa, Mexico Genealogy</title> <ns>0</ns> <id>386352</id> <revision> <id>5369276</id> <parentid>5347024</parentid> <timestamp>2023-06-14T20:58:28Z</timestamp> <contributor> <username>Amberannelarsen</username> <id>490153</id> </contributor> <minor/> <comment>Text replacement - "\&amp;#241;" to "ñ"</comment> <origin>5369276</origin> <model>wikitext</model> <format>text/x-wiki</format> <text bytes="2698" sha1="0yetzi5a03vs76bfpv273ppgniyu94p" xml:space="preserve">{{breadcrumb | link1=\[\[Mexico Genealogy|Mexico\]\] | link2=\[\[Sinaloa, Mexico Genealogy|Sinaloa\]\] | link3= | link4= | link5=\[\[Mocorito, Sinaloa, Mexico Genealogy|Mocorito\]\] }} Guide to '''Municipality of Mocorito family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records. ==History== \*En el año de 1531 con la entrada del conquistador Nuño de Guzmán al noroeste mexicano y la fundación de la villa de San Miguel de Navito, se inició la delimitación geográfica de la provincia de Culiacán. \*En 1732 cuando la expansión española llegaba más allá del río Yaqui, se encuentra el territorio dividido en provincias. \*En 1830 se decreta la separación definitiva de Sonora y Sinaloa. El nuevo estado de Sinaloa se dividió en once distritos, siendo Mocorito uno de ellos. \*Mocorito fue erigido como municipio el 8 April 1915. \*El municipio de Mocorito tiene una población de aproximadamente 45.000 personas.\&lt;ref\&gt;Wikipedia contributors, “Municipio de Mocorito” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Mocorito. accessed 26 February2021.\&lt;/ref\&gt; ==Localities within Mocorito== {| style="width:100%; vertical-align:top;" |- | \&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"\&gt; \&lt;li\&gt;Pericos\&lt;/li\&gt; \&lt;li\&gt;Mocorito\&lt;/li\&gt; \&lt;li\&gt;Caimanero\&lt;/li\&gt; \&lt;li\&gt;Melchor Ocampo\&lt;/li\&gt; \&lt;li\&gt;Recoveco\&lt;/li\&gt; \&lt;li\&gt;Higuera de los Vega\&lt;/li\&gt; \&lt;li\&gt;Potrero de los Sánchez (Estación Techa)\&lt;/li\&gt; \&lt;li\&gt;Cerro Agudo\&lt;/li\&gt; \&lt;li\&gt;El Valle de Leyva Solano (El Valle)\&lt;/li\&gt; \&lt;li\&gt;Rancho Viejo\&lt;/li\&gt; \&lt;/ul\&gt; |} ==Civil Registration== \*'''1865-1929''' {{FHL|2819522|title-id|disp=Mexico, Sinaloa, Mocorito, Civil Registration, 1865-1929}}(\*) at FamilySearch Catalog — images \*'''1922''' {{FHL|2819540|title-id|disp=Mexico, Sinaloa, Mocorito y Guasave, Civil Registration, 1922}}(\*) at FamilySearch Catalog — images ==Parish Records== \*'''1677-1968''' {{FHL|262334|title-id|disp=Iglesia Católica. Purísima Concepción (Mocorito, Sinaloa) Parish Records, 1677-1968}}(\*) at FamilySearch Catalog — images \*'''1856-1933''' {{FHL|589667|title-id|disp=Iglesia Católica. Nuestra Señora de las Angustias (Pericos, Sinaloa) Registros parroquiales, 1856-1933}}(\*) at FamilySearch Catalog — images ==Census== \*'''1930''' {{FHL|454789|title-id|disp=Censo de población del municipio de Mocorito, Sinaloa, 1930}}(\*) at FamilySearch Catalog — images ==Cemeteries== \*Panteon Reforma :\*Address: Mocorito \*Cementerio de Buena Vista :\*Address: Mocorito \*Cementerio de El Queso :\*Address: Boca de Arroyo, Mocorito ==References== \&lt;references/\&gt; \&lt;br\&gt;\&lt;br\&gt; \[\[es:Mocorito, Sinaloa, Mexico Genealogy\]\] \[\[Category:Sinaloa, Mexico\]\]</text> <sha1>0yetzi5a03vs76bfpv273ppgniyu94p</sha1> </revision> </page> <page> <title>Sinaloa, Sinaloa, Mexico Genealogy</title> <ns>0</ns> <id>386353</id> <revision> <id>5348610</id> <parentid>5348590</parentid> <timestamp>2023-05-31T20:45:17Z</timestamp> <contributor> <username>Amberannelarsen</username> <id>490153</id> </contributor> <minor/> <comment>Text replacement - "\&amp;#243;" to "ó"</comment> <origin>5348610</origin> <model>wikitext</model> <format>text/x-wiki</format> <text bytes="2313" sha1="gqx35x7qm1axjze8fhizfy6n6iasv3l" xml:space="preserve">{{breadcrumb | link1=\[\[Mexico Genealogy|Mexico\]\] | link2=\[\[Sinaloa, Mexico Genealogy|Sinaloa\]\] | link3= | link4= | link5=\[\[Sinaloa, Sinaloa, Mexico Genealogy|Sinaloa\]\] }} Guide to '''Municipality of Sinaloa family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records. ==History== \*Sinaloa de Leyva se fundó el 30 April 1583 con el nombre de Villa de San Phelipe y Santiago de Sinaloa. \*En 1732 La Villa es designada capital de la gobernación de Sinaloa. \*Sinaloa fue erigido como municipio el 25 March 1915. \*El municipio de Sinaloa tiene una población de aproximadamente 89.000 personas.\&lt;ref\&gt;Wikipedia contributors, “Municipio de Sinaloa” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Sinaloa. accessed 26 February2021.\&lt;/ref\&gt; ==Localities within Sinaloa== {| style="width:100%; vertical-align:top;" |- | \&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"\&gt; \&lt;li\&gt;Estación Naranjo\&lt;/li\&gt; \&lt;li\&gt;Sinaloa de Leyva\&lt;/li\&gt; \&lt;li\&gt;Genaro Estrada\&lt;/li\&gt; \&lt;li\&gt;Gabriel Leyva Velázquez\&lt;/li\&gt; \&lt;li\&gt;Ruiz Cortines Número Tres\&lt;/li\&gt; \&lt;li\&gt;Alfonso G. Calderón Velarde\&lt;/li\&gt; \&lt;li\&gt;Cubiri de Portelas\&lt;/li\&gt; \&lt;li\&gt;Ejido el Maquipo\&lt;/li\&gt; \&lt;li\&gt;Llano Grande 1,540\&lt;/li\&gt; \&lt;li\&gt;Santiago de Ocoroni\&lt;/li\&gt; \&lt;/ul\&gt; |} ==Civil Registration== \*'''1861-1929''' {{FHL|2819523|title-id|disp=Mexico, Sinaloa, Sinaloa, Civil Registration, 1861-1929}}(\*) at FamilySearch Catalog — images ==Parish Records== \*'''1852-1968''' {{FHL|263710|title-id|disp=Iglesia Católica. San Felipe y Santiago (Sinaloa, Sinaloa) Parish Records, 1852-1968}}(\*) at FamilySearch Catalog — images ==Census== \*'''1930''' {{FHL|454801|title-id|disp=Censo de población del municipio de Sinaloa, Sinaloa, 1930}}(\*) at FamilySearch Catalog — images ==Cemeteries== \*Panteón Municipal de Estación Naranjo Sinaloa Jesus Parra Gerardo :\*Address: Francisco Villa #0, Estación Naranjo, Sinaloa \*Cementerio Municipal :\*Address: Sinaloa Guasave, Sinaloa de Leyva, Sinaloa \*Panteón Municipal :\*Address: Isauro Vallejo #0, Tierra Blanca, Sinaloa :\*\[https://www.findagrave.com/cemetery-browse/Mexico/Sinaloa/Sinaloa-Municipality?id=county\_13465 Find a Grave\] ==References== \&lt;references/\&gt; \&lt;br\&gt;\&lt;br\&gt; \[\[es:Sinaloa, Sinaloa, Mexico Genealogy\]\] \[\[Category:Sinaloa, Mexico\]\]</text> <sha1>gqx35x7qm1axjze8fhizfy6n6iasv3l</sha1> </revision> </page></mediawiki>

Then I can manually go through and copy everything between xml:space="preserve"> and </text> to get three separate pages:

{{breadcrumb | link1=[[Mexico Genealogy|Mexico]]
| link2=[[Sinaloa, Mexico Genealogy|Sinaloa]]
| link3=
| link4=
| link5=[[Cosalá, Sinaloa, Mexico Genealogy|Cosalá]]
}}
Guide to '''Municipality of Cosalá family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records.
==History==
*El territorio donde actualmente se ubica Cosalá, estuvo ocupado por pueblos prehispánicos que se asentaron principalmente en la rivera de los ríos, como lo fueron
los grupos indígenas Tepehuanes, Acaxees y Xiximes.
*El municipio de Cosalá fue fundado el 13 March 1562.
*El municipio de Cosalá tiene una población de aproximadamente 17.000 personas.&lt;ref&gt;Wikipedia contributors, “Municipio de Cosalá” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Cosal%C3%A1. accessed 25 February2021.&lt;/ref&gt;
==Localities within Cosalá==
{| style="width:100%; vertical-align:top;"
|- |
&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"&gt;
&lt;li&gt;Cosalá&lt;/li&gt;
&lt;li&gt;El Rodeo&lt;/li&gt;
&lt;li&gt;La Llama&lt;/li&gt;
&lt;/ul&gt;
|}
==Civil Registration==
*'''1867-1929''' {{FHL|2819510|title-id|disp=Mexico, Sinaloa, Cosalá, Civil Registration, 1867-1929}}(*) at FamilySearch Catalog — images
==Parish Records==
*'''1777-1966''' {{FHL|263768|title-id|disp=Iglesia Católica. Santa Ursula (Cosala, Sinaloa) Parish Records, 1777-1966}}(*) at FamilySearch Catalog — images
*'''1874-1920''' {{FHL|260349|title-id|disp=Iglesia Católica. Santa Ursula (Cosalá, Sinaloa) Parish Records, 1874-1920}}(*) at FamilySearch Catalog — images
==Census==
==Cemeteries==
*Cementerio de San Juan Cosala
:*[https://www.findagrave.com/cemetery-browse/Mexico/Sinaloa/Cosal%C3%A1-Municipality?id=county\_13453 Find a Grave]
==References==
&lt;references/&gt;
&lt;br&gt;&lt;br&gt;
[[es:Cosalá, Sinaloa, Mexico Genealogy]]
[[Category:Sinaloa, Mexico]]

{{breadcrumb | link1=[[Mexico Genealogy|Mexico]]
| link2=[[Sinaloa, Mexico Genealogy|Sinaloa]]
| link3=
| link4=
| link5=[[Mocorito, Sinaloa, Mexico Genealogy|Mocorito]]
}}
Guide to '''Municipality of Mocorito family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records.
==History==
*En el año de 1531 con la entrada del conquistador Nuño de Guzmán al noroeste mexicano y la fundación de la villa de San Miguel de Navito, se inició la delimitación geográfica de la provincia de Culiacán.
*En 1732 cuando la expansión española llegaba más allá del río Yaqui, se encuentra el territorio dividido en provincias.
*En 1830 se decreta la separación definitiva de Sonora y Sinaloa. El nuevo estado de Sinaloa se dividió en once distritos, siendo Mocorito uno de ellos.
*Mocorito fue erigido como municipio el 8 April 1915.
*El municipio de Mocorito tiene una población de aproximadamente 45.000 personas.&lt;ref&gt;Wikipedia contributors, “Municipio de Mocorito” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Mocorito. accessed 26 February2021.&lt;/ref&gt;
==Localities within Mocorito==
{| style="width:100%; vertical-align:top;"
|- |
&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"&gt;
&lt;li&gt;Pericos&lt;/li&gt;
&lt;li&gt;Mocorito&lt;/li&gt;
&lt;li&gt;Caimanero&lt;/li&gt;
&lt;li&gt;Melchor Ocampo&lt;/li&gt;
&lt;li&gt;Recoveco&lt;/li&gt;
&lt;li&gt;Higuera de los Vega&lt;/li&gt;
&lt;li&gt;Potrero de los Sánchez (Estación Techa)&lt;/li&gt;
&lt;li&gt;Cerro Agudo&lt;/li&gt;
&lt;li&gt;El Valle de Leyva Solano (El Valle)&lt;/li&gt;
&lt;li&gt;Rancho Viejo&lt;/li&gt;
&lt;/ul&gt;
|}
==Civil Registration==
*'''1865-1929''' {{FHL|2819522|title-id|disp=Mexico, Sinaloa, Mocorito, Civil Registration, 1865-1929}}(*) at FamilySearch Catalog — images
*'''1922''' {{FHL|2819540|title-id|disp=Mexico, Sinaloa, Mocorito y Guasave, Civil Registration, 1922}}(*) at FamilySearch Catalog — images
==Parish Records==
*'''1677-1968''' {{FHL|262334|title-id|disp=Iglesia Católica. Purísima Concepción (Mocorito, Sinaloa) Parish Records, 1677-1968}}(*) at FamilySearch Catalog — images
*'''1856-1933''' {{FHL|589667|title-id|disp=Iglesia Católica. Nuestra Señora de las Angustias (Pericos, Sinaloa) Registros
parroquiales, 1856-1933}}(*) at FamilySearch Catalog — images
==Census==
*'''1930''' {{FHL|454789|title-id|disp=Censo de población del municipio de Mocorito, Sinaloa, 1930}}(*) at FamilySearch Catalog — images
==Cemeteries==
*Panteon Reforma
:*Address: Mocorito
*Cementerio de Buena Vista
:*Address: Mocorito
*Cementerio de El Queso
:*Address: Boca de Arroyo, Mocorito
==References==
&lt;references/&gt;
&lt;br&gt;&lt;br&gt;
[[es:Mocorito, Sinaloa, Mexico Genealogy]]
[[Category:Sinaloa, Mexico]]

{{breadcrumb | link1=[[Mexico Genealogy|Mexico]]
| link2=[[Sinaloa, Mexico Genealogy|Sinaloa]]
| link3=
| link4=
| link5=[[Sinaloa, Sinaloa, Mexico Genealogy|Sinaloa]]
}}
Guide to '''Municipality of Sinaloa family history and genealogy''': birth records, marriage records, death records, census records, parish registers, and military records.
==History==
*Sinaloa de Leyva se fundó el 30 April 1583 con el nombre de Villa de San Phelipe y Santiago de Sinaloa.
*En 1732 La Villa es designada capital de la gobernación de Sinaloa.
*Sinaloa fue erigido como municipio el 25 March 1915.
*El municipio de Sinaloa tiene una población de aproximadamente 89.000 personas.&lt;ref&gt;Wikipedia contributors, “Municipio de Sinaloa” in ''Wikipedia: the Free Encyclopedia'', https://es.wikipedia.org/wiki/Municipio\_de\_Sinaloa. accessed 26 February2021.&lt;/ref&gt;
==Localities within Sinaloa==
{| style="width:100%; vertical-align:top;"
|- |
&lt;ul class="column-spacing-fullscreen" style="padding-right:5px;"&gt;
&lt;li&gt;Estación Naranjo&lt;/li&gt;
&lt;li&gt;Sinaloa de Leyva&lt;/li&gt;
&lt;li&gt;Genaro Estrada&lt;/li&gt;
&lt;li&gt;Gabriel Leyva Velázquez&lt;/li&gt;
&lt;li&gt;Ruiz Cortines Número Tres&lt;/li&gt;
&lt;li&gt;Alfonso G. Calderón Velarde&lt;/li&gt;
&lt;li&gt;Cubiri de Portelas&lt;/li&gt;
&lt;li&gt;Ejido el Maquipo&lt;/li&gt;
&lt;li&gt;Llano Grande 1,540&lt;/li&gt;
&lt;li&gt;Santiago de Ocoroni&lt;/li&gt;
&lt;/ul&gt;
|}
==Civil Registration==
*'''1861-1929''' {{FHL|2819523|title-id|disp=Mexico, Sinaloa, Sinaloa, Civil Registration, 1861-1929}}(*) at FamilySearch Catalog — images
==Parish Records==
*'''1852-1968''' {{FHL|263710|title-id|disp=Iglesia Católica. San Felipe y Santiago (Sinaloa, Sinaloa) Parish Records, 1852-1968}}(*) at FamilySearch Catalog — images
==Census==
*'''1930''' {{FHL|454801|title-id|disp=Censo de población del municipio de Sinaloa, Sinaloa, 1930}}(*) at FamilySearch Catalog — images
==Cemeteries==
*Panteón Municipal de Estación Naranjo Sinaloa Jesus Parra Gerardo
:*Address: Francisco Villa #0, Estación Naranjo, Sinaloa
*Cementerio Municipal
:*Address: Sinaloa Guasave, Sinaloa de Leyva, Sinaloa
*Panteón Municipal
:*Address: Isauro Vallejo #0, Tierra Blanca, Sinaloa
:*[https://www.findagrave.com/cemetery-browse/Mexico/Sinaloa/Sinaloa-Municipality?id=county\_13465 Find a Grave]
==References==
&lt;references/&gt;
&lt;br&gt;&lt;br&gt;
[[es:Sinaloa, Sinaloa, Mexico Genealogy]]
[[Category:Sinaloa, Mexico]]

Does anyone know an efficient way to get some sort of computer to do this? I tried having ChatGPT help me write functions for Google Sheets, but they weren't working very well for me. I also tried using regular expressions, but could still only get it to do one page at a time, and still had to manually do a lot of the work, which isn't feasible when there are 300+ pages to go through. I'm happy to try to learn something new in order to do this as it would help speed up some of our processes. I am sure something like this exists, but I don't know what. Thanks for your help!

1 Comment
2023/06/26
18:34 UTC

0

Need help on cleaning this data!!

As in the picture, there are multiple records with same headers, i want to create data which has column headers and it's values below them. I am unable to find a way out. Please help!!!

3 Comments
2023/06/12
12:50 UTC

0

Need help on cleaning this data!!

As in the picture, there are multiple records with same headers, i want to create data which has column headers and it's values below them. I am unable to find a way out. Please help!!!

1 Comment
2023/06/12
12:26 UTC

5

Textraction.ai released! Flexible entity extraction - no training needed

It can extract exact values (e.g. names, prices, dates), as well as provide ChatGPT-like semantic answers (e.g. text summary). Just describe the entities with a simple format:

  • description: a free text description of what you want to extract.
  • type: string / float / integer / string.
  • variable name: a descriptive variable name.
  • (optional) valid values: limit the output to a set of specific possible values.

Very impressive, it worked great on my data which consists of product descriptions and specs.

I like the interactive demo (https://www.textraction.ai/). The service is accessible also as an API for any commercial purpose via the RapidAPI platform: https://rapidapi.com/textractionai/api/ai-textraction

0 Comments
2023/05/28
11:11 UTC

5

Best Logic to calculate Idle time

Hello guys, in our college project we have the first task which is to cleanup the data and look for extra feature.

The data set is about bikes and stations in LA and it contains 1.7 Million Rows.

We have the following features: trans_id, start_time, start_station_id, end_time,end_station_id and bike_id.

We wanted to calculate the avg. Idle time of each station. Idle time = time between return and pick up of bike at station_id .

What would be the best logic to calculate it.

1 Comment
2023/05/22
11:09 UTC

2

What is the fastest way to change this excel date format to python datetime format?

1 Comment
2023/05/01
21:03 UTC

2

Estimating predictability of raw CSV files

Seeking opinions on a tool for evaluating dataset predictability. For small/medium datasets in csv format, the tool estimates predictability on the raw data. No need to clean it; just indicate what is the target attribute. The tool uses a robust mixed attribute classifier that does not require the sorting of attributes. Of course, it does not eliminate the process of cleaning data for better results; but it can provide an initial indication of predictability. It can also be used on a smaller sample of cleaned and raw data to get an indication on how the cleaning process improves prediction.

Details available at:

https://github.com/c4pub/misc/blob/main/notebooks/csv_dataset_eval.ipynb

0 Comments
2023/04/14
11:36 UTC

0

What is the American number format?

Hello, i’m trying to dataclean some phone numbers, whereas i do understand the EU format, I have no clue about the US format

001-377-014-0631x83215

469-229-6851x300

001-117-566-5683

Here are couple examples of the data i have, I know the country code is +1 but what is the xNNN that follows some of these numbers, it could be the way they wrote it but there's a lot of similar ones so i dont think its human error

3 Comments
2023/01/03
14:37 UTC

1

Trifecta Wrangler

Does anyone have any experience using this?

I have to do a presentation on this and show my classmates a step by step guide on how to clean a dataset.

So far I've found that the smart suggestions do most of the work for me.

Before I get into it even more, anyone have any thoughts/suggestions regarding it?

0 Comments
2022/12/03
03:02 UTC

1

what attributes would help in identifying a fraud transaction in Ethereum?

I'm using this dataset https://www.kaggle.com/datasets/rupakroy/ethereum-fraud-detection.

My task is to clean it (drop some columns) and in this dataset there is a collection of many fraud and not fraud transactions denoted by flag field.

My question is which attributes will help me identify if it is a fraud transaction or in other words calculate the flag field, how do we know if the fraud is done over ether or erc20 tokens?

I'm a student with limited knowledge please help me.🥲

0 Comments
2022/08/20
12:43 UTC

0

MLOps Community (recorded) session on new open source data prep tool

Quickly move your notebooks from research to production with no extra work!
https://www.youtube.com/watch?v=6Iyt9Wip3C4

Link to tool: https://github.com/mage-ai/mage-ai

0 Comments
2022/07/26
04:36 UTC

4

Data cleaning webinar: 07/13/2022 at 9:00AM PST

Join our CEO & Co-founder, Tommy, as he reveals our new open-source data preparation tool!

Register: https://home.mlops.community/home/events/so-fresh-and-so-data-clean-2022-07-13

See you live next Wednesday, 07/13/2022 at 9:00 AM PST

https://preview.redd.it/36t7nc3538a91.png?width=1280&format=png&auto=webp&s=b032292bb85aa181f77be0d6af0a461f913dbfa8

0 Comments
2022/07/06
20:55 UTC

5

Is data cleaning one of your pain points?

We just open-sourced the alpha version of our data cleaning tool: https://github.com/mage-ai/mage-ai

Any beta testers who would be willing to test and provide feedback?

Please send any questions or feedback to me or reply here.

Thanks for the consideration!

Demo video: https://youtu.be/cRib1zOaqWs

4 Comments
2022/06/13
18:01 UTC

Back To Top