While Importing CSV files with text fields that contain double quotes using postgres COPY

While Importing CSV files with text fields that contain double quotes using postgres COPY .
By following steps you can neglect the double quotes at the time of using postgres COPY restoration

1. connecting windows server

D:
cd D:\PostgresPlus\9.4AS\bin
psql -U enterprisedb -d Health_DC

2. Sample Data’s :

"city_name"|"pincode"|"taluka_name"|"district_name"|"state_name"
"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

3. Following copy command will neglect the double quote

2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
ERROR: character with byte sequence 0x9d in encoding "WIN1252" has no equivalent in encoding "UTF8"
CONTEXT: COPY citytemp, line 358646

You need to change the character set on your windows command line before running the script with chcp. Or in postgresql you can:

SET CLIENT_ENCODING TO 'utf8';

Before importing the file.

2ndquadrant.in=# SET CLIENT_ENCODING TO 'utf8';
SET

2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
COPY 679249

 

If  your excel sheet not contain with column name means like ‘cityname’, ‘pincode’..etc means your copy command should be below format

Example : citytemp excel contain below format data’s means COPY Format should be  neglect HEADER

"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

COPY FROM for above data format 

COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV;

For Example,

If some column not contains data means you have to use NULL values while using COPY FROM

id,name
1,nijam
2,karthick
   ,nijam
4 ,   
5,

So while copying the values you have to mention value oof null otherwise copy command will throw the error

COPY schema.table FROM 'D:\2ndquadrant\nijam.csv' delimiter ','  CSV header NULL AS 'null';

If you want to insert system date While postgres COPY, for this purpose you have to use now() function
for example data should be below format

date,district
now(),chennai

POSTGRES SEQUENCE UPDATED :
You might have noticed that after bulk inserting records using the COPY statement in PostgreSQL the sequence IDs are not getting updated for any further inserts later on, and it would throw duplicate sequence ID errors.
So you would be wondering what makes this COPY statement different that it does not update the sequences,Don’t forget to update the sequence in PostgreSQL after a COPY command.

Use Below anyone of command to update the postgres sequence after bulk insertion or after COPY TO command.

ALTER SEQUENCE ehis.citytemp_seq RESTART WITH 649328;
SELECT setval('ehis.citytemp_seq', 649328);
SELECT setval('ehis.citytemp_seq', 649328);

POSTGRES MATERIALIZED VIEW REFRESHMENT :
–Below could block other connections which are trying to read from the materialized view. this is the preferbale in production environment.

REFRESH MATERIALIZED VIEW CONCURRENTLY ehis.citytemp ;

— Below materilaized view Refreshment both are same , WITH DATA is default

refresh materialized view CONCURRENTLY ehis.citytemp with data;

refresh materialized view CONCURRENTLY ehis.citytemp;

— Refresh the bulk materialized view on particular schema

select 'REFRESH MATERIALIZED VIEW '|| schemaname||'.'||matviewname ||';' as mview from pg_matviews where schemaname ='ehis';

–To finding the materialized view whose name start with ‘city’

select * from pg_matviews where schemaname ='ehis' and matviewname like 'city%' order by matviewname;

–To finding the materialized view whose name with ‘city’

select * from pg_matviews where schemaname ='ehis' and matviewname like '%city%' order by matviewname;