I developed a pipeline in AWS where I collect CPU Temperature from my notebook via Python 3, send to AWS IoT Core using security protocols, a rule sends data to Cloud Watch and DynamoDB saves it. A Data Pipeline was created to save this DynamoDB data to S3 and I want to generate plots of this data with Quick Sight.
However, I am not being able to make Quick Sight properly read the file. The S3 file looks like this:
{"timestamp":{"s":"1526819850637"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526819976032"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526819934216"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526817845094"},"payload":{"m":{"$Temperature":{"s":"48.000"}}}}
When I use the manifest file below, Quick Sight successfully reads data but the 'parseJson' command in New Calculated field disappears, making impossible to read the JSON:
{
"fileLocations": [
{
"URIs": [
"https://s3.amazonaws.com/my-bucket2/2018-05-20-12-32-49/12345-a279-1234-2269-491212345"
]
},
{
"URIPrefixes": [
"https://s3.amazonaws.com/my-bucket2/2018-05-20-12-32-49/12345-a279-1234-2269-491212345"
]
}
],"globalUploadSettings": {
"format": "CSV","delimiter":"\n","textqualifier":"'"
}
}
Quick Sight reads the JSON as:
{{"timestamp":{"s":"1526819850637"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}}
... with no 'parseJson' command.
Data in Quick Sight has no missing values and the AWS pipeline works perfectly. What can I do ?
I found the way to make it work. Simply:
{
"fileLocations": [
{
"URIs": [
"https://s3.amazonaws.com/your-bucket2/2018-05-20-12-32-49/123456789"
]
},
{
"URIPrefixes": [
"https://s3.amazonaws.com/your-bucket2/2018-05-20-12-32-49/123456789"
]
}
],"globalUploadSettings": {"format":"JSON",
"delimiter":"\n","textqualifier":"'"
}
}
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments