Get Help

Csv@2

Csv@2 loads CSV files into an in-memory DuckDB database and runs full SQL over them. Needs the Microsoft Visual C++ Redistributable on Windows.

Connects to

CSV files. CAT loads each file into a table of an in-memory DuckDB database — once for each executing thread (by default once only) — and the queries run against those tables. The data is removed from memory after all tests are evaluated.

Example

Name
FinancialData
Technology
CSV file
Provider
Csv@2
Connection string
DataToTest\Invoices.csv
Data sources:
- Name: FinancialData
  Provider: Csv@2
  Technology: Csv
  Connection string: DataToTest\Invoices.csv

Each file becomes a table named "<data source name>"."<file name without extension>" — here "FinancialData"."Invoices".

Connection string

A path, or a list of paths, to CSV files — any of:

  • an absolute path to a CSV file;
  • a relative path to a CSV file — resolved against the directory of the .cat.yaml project file;
  • an absolute or relative path with the wildcard * — allowed only in the file name, no recursion into subdirectories;
  • several of the above, separated by commas.
Name
My CSV data
Technology
CSV file
Provider
Csv@2
Connection string
DataToTest\*.csv,
D:\Export\Financial\Invoices-2024.csv

Data sources:
- Name: My CSV data
  Provider: Csv@2
  Technology: Csv
  Connection string: >
     DataToTest\*.csv,
     D:\Export\Financial\Invoices-2024.csv

Every file that matches is loaded into a separate table; the files are not combined into one. If DataToTest holds Summary-2023.csv and Summary-2024.csv, a query can combine them itself:

SELECT * FROM "My CSV data"."Summary-2023"
UNION ALL
SELECT * FROM "My CSV data"."Summary-2024"

Settings

Setting Default Meaning
Settings auto_detect = true Options for DuckDB’s CSV reader, written as one comma-separated list — delimiter, header row, compression, quoting, types … Every option of DuckDB read_csv is accepted; CAT hands the string over as it is.
Name
FinancialData
Technology
CSV file
Provider
Csv@2
Connection string
DataToTest\Invoices.csv
Settings
delim = '|', header = true, compression = 'gzip'
Data sources:
- Name: FinancialData
  Provider: Csv@2
  Technology: Csv
  Connection string: DataToTest\Invoices.csv
  Settings: delim = '|', header = true, compression = 'gzip'

Query

One SQL statement in DuckDB’s dialect against the loaded tables — table names are "<data source name>"."<file name without extension>", quoted when they contain spaces or a hyphen. Because the data sits in a database, the whole language is available: joins, grouping, CTEs, window functions, functions for dates, strings and numbers. What that looks like in practice — SELECT, joins and set operations, expressions, functions — is in Query CSV and Excel data with DuckDB SQL; the full reference is the DuckDB SQL documentation.

Prerequisites

DuckDB ships with CAT; Windows and Linux alike. Two things must be in place on the machine:

Microsoft Visual C++ Redistributable (Windows)

A very common prerequisite used by much other software, so it is probably already installed. To verify or install it, go to Latest Microsoft Visual C++ Redistributable Version and download and install the “X64” version:

Microsoft Visual C++ Redistributable web download link

If the installer shows a dialog with “Repair”, “Uninstall” and “Close” buttons, just close it — the prerequisite is already there. Otherwise install it (next, next, next…).

Internet connection

DuckDB loads its extensions from http://extensions.duckdb.org. The URL has to be reachable — connect to the Internet and white-list it if needed. If you need to test CSV data offline, file a request using the “Get Help” button on the right of this site.

Limitations

  • Csv@2 cannot serve lists of test or data source definitions; Csv@1 can.
  • Needs access to extensions.duckdb.org at start — see Prerequisites.

Serving test and data source definitions

Csv@2 cannot serve lists of definitions; use Csv@1 for that.

← Csv@1
Dax@2 →