<p><strong>Summary</strong></p>
<p>GitTables 1M (<a href="https://gittables.github.io/">https://gittables.github.io</a>) is a corpus of currently 1M relational tables extracted from CSV files in GitHub repositories, that are associated with a license that allows distribution. We aim to grow this to at least 10M tables.</p>
<p>Each parquet file in this corpus represents a table with the original content (e.g. values and header) as extracted from the corresponding CSV file. Table columns are enriched with annotations corresponding to >2K semantic types from Schema.org and DBpedia (provided as metadata of the parquet file). These column annotations consist of, for example, semantic types, hierarchical relations to other types, and descriptions.</p>
<p>We believe GitTables can facilitate many use-cases, among which:</p>
<ul>
<li>
<p>Data integration, search and validation.</p>
</li>
<li>
<p>Data visualization and analysis recommendation.</p>
</li>
<li>
<p>Schema analysis and completion for e.g. database or knowledge base design.</p>
</li>
</ul>
<p>If you have questions, the paper, documentation, and contact details are provided on the website: <a href="https://gittables.github.io/">https://gittables.github.io</a>. We recommend using Zenodo's API to easily download the full dataset (i.e. all zipped topic subsets).</p>
<p> </p>
<p><strong>Dataset contents</strong></p>
<p>The data is provided in subsets of tables stored in parquet files, each subset corresponds to a term that was used to query GitHub with. The column annotations and other metadata (e.g. URL and repository license) are attached to the metadata of the parquet file. This version corresponds to this version of the paper <a href="https://arxiv.org/abs/2106.07258v4">https://arxiv.org/abs/2106.07258v4</a>.</p>
<p>In summary, this dataset can be characterized as follows:</p>
<table>
<tbody>
<tr>
<td>
<p><strong>Statistic</strong></p>
</td>
<td>
<p><strong>Value</strong></p>
</td>
</tr>
<tr>
<td>
<p># tables</p>
</td>
<td>
<p>1M</p>
</td>
</tr>
<tr>
<td>
<p>average # columns</p>
</td>
<td>
<p>12</p>
</td>
</tr>
<tr>
<td>
<p>average # rows</p>
</td>
<td>
<p>142</p>
</td>
</tr>
<tr>
<td>
<p># annotated tables (at least 1 column annotation)</p>
</td>
<td>
<p>723K+ (DBpedia), 738K+ (Schema.org)</p>
</td>
</tr>
<tr>
<td>
<p># unique semantic types</p>
</td>
<td>
<p>835 (DBpedia), 677 (Schema.org)</p>
</td>
</tr>
</tbody>
</table>
<p> </p>
<p><strong>Future releases</strong></p>
<p>Future releases will include the following:</p>
<ul>
<li>
<p>Increased number of tables (expected at least 10M)</p>
</li>
</ul>
<p> </p>
<p><strong>Associated datasets</strong></p>
<p>- GitTables benchmark - column type detection: <a href="https://zenodo.org/record/5706316">https://zenodo.org/record/5706316</a></p>
<p>- GitTables 1M - CSV files: <a href="https://zenodo.org/record/6515973">https://zenodo.org/record/6515973</a></p>
<p> </p>