CSV - Revision history

Jkuehlthau: /* Unable to translate bytes [xx] at index yyy from specified code page to Unicode. */

2025-09-05T18:35:49Z

Unable to translate bytes [xx] at index yyy from specified code page to Unicode.

← Older revision		Revision as of 18:35, 5 September 2025
Line 35:		Line 35:

	==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==		==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==
	This means there is an invalid/unknown/unexpected character(s) in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.		This means there is an invalid/unknown/unexpected character(s) in the Excel file. I use VS Code, https://code.visualstudio.com/, to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.

	Another time the above did NOT resolve the issue for me. I had to use VS Code to Find the non-ascii character with regular expressions. This should be possible in many text editors. I searched for this:<br>		Another time the above did NOT resolve the issue for me. I had to use VS Code to Find the non-ascii character with regular expressions. This should be possible in many text editors. I searched for this:<br>

Jkuehlthau: /* Unable to translate bytes [xx] at index yyy from specified code page to Unicode. */

2023-08-04T15:48:54Z

Unable to translate bytes [xx] at index yyy from specified code page to Unicode.

← Older revision		Revision as of 15:48, 4 August 2023
Line 36:		Line 36:
	==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==		==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==
	This means there is an invalid/unknown/unexpected character(s) in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.		This means there is an invalid/unknown/unexpected character(s) in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.

			Another time the above did NOT resolve the issue for me. I had to use VS Code to Find the non-ascii character with regular expressions. This should be possible in many text editors. I searched for this:<br>
			[^\x00-\x7F]+
			<br>
			This may also work:<br>
			([^\x00-\x7F])

Jkuehlthau: /* Origin */

2023-03-09T16:45:14Z

Origin

← Older revision		Revision as of 16:45, 9 March 2023
Line 1:		Line 1:
	=Origin=		=Origin=
	'Do not have multiple columns in your CSV file with the same column header. The file will NOT work if column headers are duplicated.'		'''Do not have multiple columns in your CSV file with the same column header. The file will NOT work if column headers are duplicated.'''

	We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:		We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:

Jkuehlthau: /* Origin */

2023-03-09T16:44:53Z

Origin

← Older revision		Revision as of 16:44, 9 March 2023
Line 1:		Line 1:
	=Origin=		=Origin=
			'Do not have multiple columns in your CSV file with the same column header. The file will NOT work if column headers are duplicated.'

	We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:		We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:
	<source lang="xml">		<source lang="xml">

Jkuehlthau: /* Unable to translate bytes [xx] at index yyy from specified code page to Unicode. */

2023-01-13T18:33:25Z

Unable to translate bytes [xx] at index yyy from specified code page to Unicode.

← Older revision		Revision as of 18:33, 13 January 2023
Line 33:		Line 33:

	==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==		==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==
	This means there is an invalid character in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.		This means there is an invalid/unknown/unexpected character(s) in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.

Jkuehlthau: /* Issues */

2023-01-13T18:33:02Z

Issues

← Older revision		Revision as of 18:33, 13 January 2023
Line 31:		Line 31:

	The download is labelled "Microsoft Access Database Engine 2010 Redistributable" and I found it here: https://www.microsoft.com/en-us/download/details.aspx?id=13255.		The download is labelled "Microsoft Access Database Engine 2010 Redistributable" and I found it here: https://www.microsoft.com/en-us/download/details.aspx?id=13255.

			==Unable to translate bytes [xx] at index yyy from specified code page to Unicode.==
			This means there is an invalid character in the Excel file. I use VS Code to search the file for the "�" character and then replace them with an empty string: "". This resolved the issue for me.

Jkuehlthau: /* Origin */

2022-10-27T20:43:41Z

Origin

← Older revision		Revision as of 20:43, 27 October 2022
Line 2:		Line 2:
	We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:		We've seen some display issues with the CSV connector. The Connection String sample is not quite correct. It should read:
	<source lang="xml">		<source lang="xml">
	Data Source="C:\Starfish\Folder";Extended Properties="Text;HDR=Yes;"		Data Source="C:\Starfish\Folder";Extended Properties="Text;HDR=Yes;TypeDetectionScheme=None;"
	</source>		</source>
	Your SqL Selection Statement then selects from the filename:		Your SqL Selection Statement then selects from the filename:

Jkuehlthau: /* Issues */

2018-12-20T22:47:28Z

Issues

← Older revision		Revision as of 22:47, 20 December 2018
Line 27:		Line 27:
	You should now see a schema.ini file in the directory. You can edit this file using a text editor to modify the expected data type. For example, for varchar, set the column to Char Width 255		You should now see a schema.ini file in the directory. You can edit this file using a text editor to modify the expected data type. For example, for varchar, set the column to Char Width 255
	You will need to remove the line CharacterSet = OEM from the schema.ini file if present. Also remove the values for any other files in the same directory for which you do not wish to define the schema.		You will need to remove the line CharacterSet = OEM from the schema.ini file if present. Also remove the values for any other files in the same directory for which you do not wish to define the schema.
			==Use ODBC instead of CSV?==
			I had a 111MB CSV with 442,427 rows in it. The CSV Connector could not read it. I downloaded the 64-bit Microsoft ODBC CSV Engine and it read the file without issue.

			The download is labelled "Microsoft Access Database Engine 2010 Redistributable" and I found it here: https://www.microsoft.com/en-us/download/details.aspx?id=13255.

Msheehan at 18:24, 11 June 2018

2018-06-11T18:24:41Z

← Older revision		Revision as of 18:24, 11 June 2018
Line 20:		Line 20:
	==Type mismatch issue where column contains some alpha numeric entries but Starfish believes the data is numeric.==		==Type mismatch issue where column contains some alpha numeric entries but Starfish believes the data is numeric.==
	If a column of data contains numeric data for the first few rows and then later contains alphanumeric data Starfish will expect numeric data from every row in the data and will then fail to parse the alphanumeric data. Because by default the Microsoft Jet driver that is used for this functionality only scans the first 16 rows, it tries to make a ‘guess’ based on that data for each column type. This is only a problem is the data changes from numeric to alphanumeric partway through the data as thinks the data is all numbers. In registry, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Jet\4.0\Engines\Text Find the MaxScanRows key, and change the value to ‘0’. Restart Starfish. It may also be necessary to perform an IISRESET.		If a column of data contains numeric data for the first few rows and then later contains alphanumeric data Starfish will expect numeric data from every row in the data and will then fail to parse the alphanumeric data. Because by default the Microsoft Jet driver that is used for this functionality only scans the first 16 rows, it tries to make a ‘guess’ based on that data for each column type. This is only a problem is the data changes from numeric to alphanumeric partway through the data as thinks the data is all numbers. In registry, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Jet\4.0\Engines\Text Find the MaxScanRows key, and change the value to ‘0’. Restart Starfish. It may also be necessary to perform an IISRESET.
			==Value too long to parse==
			Depending on what values Starfish is expecting to see in a column, it may generate an error for values that do not match the expected data type. You can create a schema.ini file to tell Starfish what data types to expect, rather than relying on Starfish's guess based on scanning the first few rows of data.

			First, use the ODBC Data Sources tool to generate a schema file. Open the tool, go to System DSN tab, and click Add. Select Microsoft Text Driver and click Finish. Uncheck 'Use Current Directory' setting and click the Select Directory button. Navigate to the directory where your file is saved, and click OK. Then click the Options button and click the Define Format button. Select your file from the list, check the Column Name Header box, and click Guess. (You may also have to adjust Rows to Scan setting.) Click OK. Cancel out of ODBC Data Sources tool.

			You should now see a schema.ini file in the directory. You can edit this file using a text editor to modify the expected data type. For example, for varchar, set the column to Char Width 255
			You will need to remove the line CharacterSet = OEM from the schema.ini file if present. Also remove the values for any other files in the same directory for which you do not wish to define the schema.

Msheehan: /* Origin */

2018-04-18T17:49:49Z

Origin

← Older revision		Revision as of 17:49, 18 April 2018
Line 14:		Line 14:
	SELECT * FROM "districts.csv"		SELECT * FROM "districts.csv"
	</source>		</source>

			The file needs to be in a location that the Starfish app pool can read from. Either grant access to the directory for "IIS_IUSRS" or put the CSV file under C:\inetpub\wwwroot\StarfishEngine

	=Issues=		=Issues=
	==Type mismatch issue where column contains some alpha numeric entries but Starfish believes the data is numeric.==		==Type mismatch issue where column contains some alpha numeric entries but Starfish believes the data is numeric.==
	If a column of data contains numeric data for the first few rows and then later contains alphanumeric data Starfish will expect numeric data from every row in the data and will then fail to parse the alphanumeric data. Because by default the Microsoft Jet driver that is used for this functionality only scans the first 16 rows, it tries to make a ‘guess’ based on that data for each column type. This is only a problem is the data changes from numeric to alphanumeric partway through the data as thinks the data is all numbers. In registry, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Jet\4.0\Engines\Text Find the MaxScanRows key, and change the value to ‘0’. Restart Starfish. It may also be necessary to perform an IISRESET.		If a column of data contains numeric data for the first few rows and then later contains alphanumeric data Starfish will expect numeric data from every row in the data and will then fail to parse the alphanumeric data. Because by default the Microsoft Jet driver that is used for this functionality only scans the first 16 rows, it tries to make a ‘guess’ based on that data for each column type. This is only a problem is the data changes from numeric to alphanumeric partway through the data as thinks the data is all numbers. In registry, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Jet\4.0\Engines\Text Find the MaxScanRows key, and change the value to ‘0’. Restart Starfish. It may also be necessary to perform an IISRESET.