Postgres Copy Command Tutorial
PostgreSQL COPY data to Text File
In this post you can learn how to take the backup of particular table by using Postgres COPY command.
Examples:
Step1. First list down the table and and describe the table data which one want to copy from postgres database
postgres=# select * from batch;
batch_id | batch_name | members
----------+-------------------------------------+---------
1 | nijam | 1
2 | nijam | 5
3 | nijam | 5
4 | nijam | 5
4 | nijam | 8
(5 rows)Step2. then give the copy query which will copy the table from postgres server to csv file
postgres=# copy batch to '/opt/PostgreSQL/9.3/data/michel.csv' with delimiter ',' CSV header;
COPY 5step3. then read the file for copied data whether is correct or not
[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/michel.csv
batch_id,batch_name,members
1,nijam ,1
2,nijam ,5
3,nijam ,5
4,nijam ,5
4,nijam ,8Example 2.
second example is same as previous but small difference is here copied data including column name
postgres=# copy (select * from pg_tables) to '/opt/PostgreSQL/9.3/data/nijam2.csv'
with delimiter ',' CSV header;
COPY 83[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/nijam2.csv
schemaname,tablename,tableowner,tablespace,hasindexes,hasrules,hastriggers
pg_catalog,pg_statistic,postgres,,t,f,f
pg_catalog,pg_type,postgres,,t,f,f
public,t1,postgres,,f,f,f
pg_catalog,pg_authid,postgres,pg_global,t,f,f
pg_catalog,pg_proc,postgres,,t,f,f
pg_catalog,pg_class,postgres,,t,f,f
pg_catalog,pg_user_mapping,postgres,,t,f,f
pg_catalog,pg_attribute,postgres,,t,f,f
pg_catalog,pg_constraint,postgres,,t,f,f
pg_catalog,pg_inherits,postgres,,t,f,f
pg_catalog,pg_index,postgres,,t,f,f
pg_catalog,pg_operator,postgres,,t,f,f
pg_catalog,pg_opfamily,postgres,,t,f,f
pg_catalog,pg_opclass,postgres,,t,f,f
pg_catalog,pg_am,postgres,,t,f,f
pg_catalog,pg_amop,postgres,,t,f,f
pg_catalog,pg_amproc,postgres,,t,f,f
pg_catalog,pg_language,postgres,,t,f,f
pg_catalog,pg_largeobject_metadata,postgres,,t,f,f
pg_catalog,pg_aggregate,postgres,,t,f,f
pg_catalog,pg_database,postgres,pg_global,t,f,f
pg_catalog,pg_trigger,postgres,,t,f,f
pg_catalog,pg_event_trigger,postgres,,t,f,f
pg_catalog,pg_description,postgres,,t,f,f
pg_catalog,pg_cast,postgres,,t,f,f
pg_catalog,pg_enum,postgres,,t,f,f
pg_catalog,pg_conversion,postgres,,t,f,f
pg_catalog,pg_depend,postgres,,t,f,f
pg_catalog,pg_db_role_setting,postgres,pg_global,t,f,f
pg_catalog,pg_tablespace,postgres,pg_global,t,f,f
pg_catalog,pg_pltemplate,postgres,pg_global,t,f,f
pg_catalog,pg_auth_members,postgres,pg_global,t,f,f
pg_catalog,pg_shdepend,postgres,pg_global,t,f,f
pg_catalog,pg_shdescription,postgres,pg_global,t,f,f
pg_catalog,pg_ts_config,postgres,,t,f,f
pg_catalog,pg_ts_config_map,postgres,,t,f,f
pg_catalog,pg_ts_dict,postgres,,t,f,f
pg_catalog,pg_ts_parser,postgres,,t,f,f
pg_catalog,pg_ts_template,postgres,,t,f,f
pg_catalog,pg_extension,postgres,,t,f,f
pg_catalog,pg_foreign_data_wrapper,postgres,,t,f,f
pg_catalog,pg_foreign_server,postgres,,t,f,f
pg_catalog,pg_foreign_table,postgres,,t,f,f
pg_catalog,pg_default_acl,postgres,,t,f,f
pg_catalog,pg_seclabel,postgres,,t,f,f
pg_catalog,pg_shseclabel,postgres,pg_global,t,f,f
pg_catalog,pg_collation,postgres,,t,f,f
pg_catalog,pg_range,postgres,,t,f,f
pg_catalog,pg_largeobject,postgres,,t,f,f
information_schema,sql_implementation_info,postgres,,f,f,f
information_schema,sql_languages,postgres,,f,f,f
information_schema,sql_packages,postgres,,f,f,f
information_schema,sql_parts,postgres,,f,f,f
public,postgres,postgres,,t,f,f
information_schema,sql_sizing_profiles,postgres,,f,f,f
pg_catalog,pg_attrdef,postgres,,t,f,f
public,primtab,u8,,t,f,f
public,films,u8,,t,f,f
public,distributors,u8,,t,f,f
pg_catalog,pg_namespace,postgres,,t,f,f
pg_catalog,pg_rewrite,postgres,,t,f,f
public,k,u8,,t,f,f
public,distributorc,u8,,f,f,f
public,distributorsct,u8,,f,f,f
public,postgres1,postgres,,t,f,f
public,postgres2,postgres,,t,f,f
public,postgres3,postgres,,t,f,f
public,postgres4,postgres,,t,f,f
public,postgres_5,postgres,,t,f,f
public,students,postgres,,t,f,f
public,india,postgres,,t,f,f
information_schema,sql_features,postgres,,f,f,f
information_schema,sql_sizing,postgres,,f,f,f
public,batch,postgres,,t,f,f
public,cellphones,postgres,,t,f,f
public,mobiles,postgres,,t,f,f
public,location,postgres,,t,f,f
public,dept,postgres,,f,f,f
public,products1,postgres,,f,f,f
public,products2,postgres,,f,f,f
public,t7,postgres,,f,f,f
public,village,postgres,,f,f,f
public,emp_d,postgres,,f,f,fPostgreSQL COPY data From Text File
Examples:
Step1. create the text file and insert some data into text file.
[root@p1 ~]# vi /opt/PostgreSQL/9.3/data/new.txt
[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/new.txt
1,akash
2,varun
3,makash
4,nijam
5,benzstep2. create a table as per text file structure here i created the table with to column because in text file having 2 column one is number which you can consider id with integer data type second is name which is name with text data type
postgres=# create table dept(id integer,name text);
CREATE TABLEstep3. Now you can issue the query which will copy the data from text file to database
postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
ERROR: could not open file "/opt/PostgreSQL/9.3/data/new.txt" for writing: Permission deniedBecause new.txt file is root user file so you have to change the file ownership to postgres user
Solution:
[root@p1 ~]# chown postgres:postgres /opt/PostgreSQL/9.3/data/new.txt
[root@p1 ~]# cd /opt/PostgreSQL/9.3/data/
[root@p1 data]# ls -lrt new.txt
-rw-r--r-- 1 postgres postgres 43 Apr 3 16:44 new.txt
postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
COPY 5
postgres=# select * from dept;
id | name
----+--------
1 | akash
2 | varun
3 | makash
4 | nijam
5 | benz
(5 rows)Example 2. this is same as previous example but only thing is here added only one extra column which is date
[root@p1 data]# vi nijam.csv
[root@p1 data]# cat nijam.csv
1, mala,20-SEP-90 00:00:00
2, tarun,10-OCT-08 00:00:00
3, lara,12-JAN-09 00:00:00
4, sara,08-MAY-01 00:00:00
5, guttu,10-MAY-09 00:00:00
[root@p1 data]# ls -lrt nijam.csv
-rw-r--r-- 1 root root 174 Apr 3 17:22 nijam.csv
-- check the file permission file ownership permission should be postgres user[root@p1data]# pwd
/opt/PostgreSQL/9.3/data/
[root@p1 data]# chown postgres:postgres /opt/PostgreSQL/9.3/data/nijam.csv
[root@p1 data]# ls -lrt nijam.csv
-rw-r--r-- 1 postgres postgres 174 Apr 3 17:22 nijam.csv— then create the table structure as per data structures then import the data into postgresql server
postgres=# create table emp_d(id integer,name text, dob date);
CREATE TABLE
postgres=# copy emp_d from '/opt/PostgreSQL/9.3/data/nijam.csv' with delimiter ',';
COPY 5
postgres=# select * from emp_d;
id | name | dob
----+-------------+------------
1 | mala | 1990-09-20
2 | tarun | 2008-10-10
3 | lara | 2009-01-12
4 | sara | 2001-05-08
5 | guttu | 2009-05-10
(5 rows)More Example postgres COPY command
Below Examples are worked on postgres windows server so you can use ‘COPY ‘ instead of using ‘ \COPY ‘.
While Importing CSV files with text fields that contain double quotes using postgres COPY .
By using below copy command you can restore the postgres data into the target table if data has been double quotes as well.
1. connecting windows server
D:
cd D:\PostgresPlus\9.4AS\bin
psql -U enterprisedb -d Health_DC2. Sample Data’s :
"city_name"|"pincode"|"taluka_name"|"district_name"|"state_name"
"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"3. Following copy command will neglect the double quote
2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
ERROR: character with byte sequence 0x9d in encoding "WIN1252" has no equivalent in encoding "UTF8"
CONTEXT: COPY citytemp, line 358646You need to change the character set on your windows command line before running the script with chcp. Or in postgresql you can:
SET CLIENT_ENCODING TO 'utf8';Before importing the file.
2ndquadrant.in=# SET CLIENT_ENCODING TO 'utf8';
SET
2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
COPY 679249
If your excel sheet not contain with column name means like ‘cityname’, ‘pincode’..etc means your copy command should be below format
Example : citytemp excel contain below format data’s means COPY Format should be neglect HEADER
"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
COPY FROM for above data format
COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV;For Example,
If some column not contains data means you have to use NULL values while using COPY FROM
id,name
1,nijam
2,karthick
,nijam
4 ,
5,So while copying the values you have to mention value oof null otherwise copy command will throw the error
COPY schema.table FROM 'D:\2ndquadrant\nijam.csv' delimiter ',' CSV header NULL AS 'null';If you want to insert system date While postgres COPY, for this purpose you have to use now() function
for example data should be below format
date,district
now(),chennaiIS SEQUENCE UPDATING IMPORTANT ?
You might have noticed that after bulk inserting records using the COPY statement in PostgreSQL the sequence IDs are not getting updated for any further inserts later on, and it would throw duplicate sequence ID errors.
So you would be wondering what makes this COPY statement different that it does not update the sequences,Don’t forget to update the sequence in PostgreSQL after a COPY command.
Use Below anyone of command to update the postgres sequence after bulk insertion or after COPY TO command.
ALTER SEQUENCE ehis.citytemp_seq RESTART WITH 649328;
SELECT setval('ehis.citytemp_seq', 649328);
SELECT setval('ehis.citytemp_seq', 649328);WHY NEED TO REFRESH POSTGRES MATERIALIZED VIEW ?
–Below could block other connections which are trying to read from the materialized view. this is the preferbale in production environment.
REFRESH MATERIALIZED VIEW CONCURRENTLY ehis.citytemp ;— Below materilaized view Refreshment both are same , WITH DATA is default
refresh materialized view CONCURRENTLY ehis.citytemp with data;
refresh materialized view CONCURRENTLY ehis.citytemp;— Refresh the bulk materialized view on particular schema
select 'REFRESH MATERIALIZED VIEW '|| schemaname||'.'||matviewname ||';' as mview from pg_matviews where schemaname ='ehis';–To finding the materialized view whose name start with ‘city’
select * from pg_matviews where schemaname ='ehis' and matviewname like 'city%' order by matviewname;–To finding the materialized view whose name with ‘city’
select * from pg_matviews where schemaname ='ehis' and matviewname like '%city%' order by matviewname;
Login with Google