Postgres Copy Command Tutorial

PostgreSQL COPY data to Text File

In this post you can learn how to take the backup of particular table by using Postgres COPY command.

Examples:
Step1. First list down the table and and describe the table data  which one want to copy from postgres database

postgres=# select * from batch;
 batch_id |             batch_name              | members 
----------+-------------------------------------+---------
        1 | nijam                               |       1
        2 | nijam                               |       5
        3 | nijam                               |       5
        4 | nijam                               |       5
        4 | nijam                               |       8
(5 rows)

Step2. then give the copy query which  will copy the table from postgres server to csv file

postgres=# copy batch to '/opt/PostgreSQL/9.3/data/michel.csv' with delimiter ',' CSV header;
COPY 5

step3.  then read the file for copied data whether is correct or not

[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/michel.csv
batch_id,batch_name,members
1,nijam                              ,1
2,nijam                              ,5
3,nijam                              ,5
4,nijam                              ,5
4,nijam                              ,8

Example 2.

second example is same as previous but small difference is here copied data including column name

postgres=# copy (select * from pg_tables) to '/opt/PostgreSQL/9.3/data/nijam2.csv' 
with delimiter ',' CSV header; 
COPY 83
[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/nijam2.csv
schemaname,tablename,tableowner,tablespace,hasindexes,hasrules,hastriggers
pg_catalog,pg_statistic,postgres,,t,f,f
pg_catalog,pg_type,postgres,,t,f,f
public,t1,postgres,,f,f,f
pg_catalog,pg_authid,postgres,pg_global,t,f,f
pg_catalog,pg_proc,postgres,,t,f,f
pg_catalog,pg_class,postgres,,t,f,f
pg_catalog,pg_user_mapping,postgres,,t,f,f
pg_catalog,pg_attribute,postgres,,t,f,f
pg_catalog,pg_constraint,postgres,,t,f,f
pg_catalog,pg_inherits,postgres,,t,f,f
pg_catalog,pg_index,postgres,,t,f,f
pg_catalog,pg_operator,postgres,,t,f,f
pg_catalog,pg_opfamily,postgres,,t,f,f
pg_catalog,pg_opclass,postgres,,t,f,f
pg_catalog,pg_am,postgres,,t,f,f
pg_catalog,pg_amop,postgres,,t,f,f
pg_catalog,pg_amproc,postgres,,t,f,f
pg_catalog,pg_language,postgres,,t,f,f
pg_catalog,pg_largeobject_metadata,postgres,,t,f,f
pg_catalog,pg_aggregate,postgres,,t,f,f
pg_catalog,pg_database,postgres,pg_global,t,f,f
pg_catalog,pg_trigger,postgres,,t,f,f
pg_catalog,pg_event_trigger,postgres,,t,f,f
pg_catalog,pg_description,postgres,,t,f,f
pg_catalog,pg_cast,postgres,,t,f,f
pg_catalog,pg_enum,postgres,,t,f,f
pg_catalog,pg_conversion,postgres,,t,f,f
pg_catalog,pg_depend,postgres,,t,f,f
pg_catalog,pg_db_role_setting,postgres,pg_global,t,f,f
pg_catalog,pg_tablespace,postgres,pg_global,t,f,f
pg_catalog,pg_pltemplate,postgres,pg_global,t,f,f
pg_catalog,pg_auth_members,postgres,pg_global,t,f,f
pg_catalog,pg_shdepend,postgres,pg_global,t,f,f
pg_catalog,pg_shdescription,postgres,pg_global,t,f,f
pg_catalog,pg_ts_config,postgres,,t,f,f
pg_catalog,pg_ts_config_map,postgres,,t,f,f
pg_catalog,pg_ts_dict,postgres,,t,f,f
pg_catalog,pg_ts_parser,postgres,,t,f,f
pg_catalog,pg_ts_template,postgres,,t,f,f
pg_catalog,pg_extension,postgres,,t,f,f
pg_catalog,pg_foreign_data_wrapper,postgres,,t,f,f
pg_catalog,pg_foreign_server,postgres,,t,f,f
pg_catalog,pg_foreign_table,postgres,,t,f,f
pg_catalog,pg_default_acl,postgres,,t,f,f
pg_catalog,pg_seclabel,postgres,,t,f,f
pg_catalog,pg_shseclabel,postgres,pg_global,t,f,f
pg_catalog,pg_collation,postgres,,t,f,f
pg_catalog,pg_range,postgres,,t,f,f
pg_catalog,pg_largeobject,postgres,,t,f,f
information_schema,sql_implementation_info,postgres,,f,f,f
information_schema,sql_languages,postgres,,f,f,f
information_schema,sql_packages,postgres,,f,f,f
information_schema,sql_parts,postgres,,f,f,f
public,postgres,postgres,,t,f,f
information_schema,sql_sizing_profiles,postgres,,f,f,f
pg_catalog,pg_attrdef,postgres,,t,f,f
public,primtab,u8,,t,f,f
public,films,u8,,t,f,f
public,distributors,u8,,t,f,f
pg_catalog,pg_namespace,postgres,,t,f,f
pg_catalog,pg_rewrite,postgres,,t,f,f
public,k,u8,,t,f,f
public,distributorc,u8,,f,f,f
public,distributorsct,u8,,f,f,f
public,postgres1,postgres,,t,f,f
public,postgres2,postgres,,t,f,f
public,postgres3,postgres,,t,f,f
public,postgres4,postgres,,t,f,f
public,postgres_5,postgres,,t,f,f
public,students,postgres,,t,f,f
public,india,postgres,,t,f,f
information_schema,sql_features,postgres,,f,f,f
information_schema,sql_sizing,postgres,,f,f,f
public,batch,postgres,,t,f,f
public,cellphones,postgres,,t,f,f
public,mobiles,postgres,,t,f,f
public,location,postgres,,t,f,f
public,dept,postgres,,f,f,f
public,products1,postgres,,f,f,f
public,products2,postgres,,f,f,f
public,t7,postgres,,f,f,f
public,village,postgres,,f,f,f
public,emp_d,postgres,,f,f,f

PostgreSQL COPY data From Text File

Here i’m Explained about How to insert the data into postgres database from text file.

Examples:
insert the data into the database from text file 

Step1. create the text file  and insert some data into text file.
[root@p1 ~]# vi /opt/PostgreSQL/9.3/data/new.txt
[root@p1 ~]# cat /opt/PostgreSQL/9.3/data/new.txt
1,akash
2,varun
3,makash
4,nijam
5,benz

step2. create a table as per text file structure here i created the table with to column  because in text file having 2 column one is number which you can consider id with integer data type second is name which is name with text data type

postgres=# create table dept(id integer,name text);
CREATE TABLE

step3. Now you can issue the query  which will copy the data from text file to database

postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
ERROR:  could not open file "/opt/PostgreSQL/9.3/data/new.txt" for writing: Permission denied

Because new.txt file is root user file so you have to change the file ownership to postgres user
Solution:

[root@p1 ~]# chown postgres:postgres /opt/PostgreSQL/9.3/data/new.txt
[root@p1 ~]# cd /opt/PostgreSQL/9.3/data/

[root@p1 data]# ls -lrt new.txt 
-rw-r--r-- 1 postgres postgres 43 Apr  3 16:44 new.txt

postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
COPY 5
postgres=# select * from dept;
 id |  name  
----+--------
  1 | akash
  2 | varun
  3 | makash
  4 | nijam
  5 | benz
(5 rows)

Example 2. this is same as previous  example but only thing is  here added only one extra column which is date

— check the contents of nijam.csv file which will be helpful to creating table structure
[root@p1 data]# vi nijam.csv

[root@p1 data]# cat nijam.csv 
1,      mala,20-SEP-90 00:00:00
2,      tarun,10-OCT-08 00:00:00
3,      lara,12-JAN-09 00:00:00
4,      sara,08-MAY-01 00:00:00
5,      guttu,10-MAY-09 00:00:00

[root@p1 data]# ls -lrt nijam.csv 
-rw-r--r-- 1 root root 174 Apr  3 17:22 nijam.csv

-- check the file permission file ownership permission should be postgres user
[root@p1data]# pwd
/opt/PostgreSQL/9.3/data/

[root@p1 data]# chown postgres:postgres /opt/PostgreSQL/9.3/data/nijam.csv 
[root@p1 data]# ls -lrt nijam.csv 
-rw-r--r-- 1 postgres postgres 174 Apr  3 17:22 nijam.csv

— then create the table structure as per data structures then import the data into postgresql server

postgres=# create table emp_d(id integer,name text, dob date);
CREATE TABLE

postgres=# copy emp_d from '/opt/PostgreSQL/9.3/data/nijam.csv' with delimiter ',';
COPY 5
postgres=# select * from emp_d;
 id |    name     |    dob     
----+-------------+------------
  1 |       mala  | 1990-09-20
  2 |       tarun | 2008-10-10
  3 |       lara  | 2009-01-12
  4 |       sara  | 2001-05-08
  5 |       guttu | 2009-05-10
(5 rows)

More Example postgres COPY command

Below  Examples are worked on postgres windows server so you can  use ‘COPY ‘ instead of using ‘ \COPY ‘.

While Importing CSV files with text fields that contain double quotes using postgres COPY .

By using below copy command you can restore the postgres data into the target table if data has been double quotes as well.

1. connecting windows server

D:
cd D:\PostgresPlus\9.4AS\bin
psql -U enterprisedb -d Health_DC

2. Sample Data’s :

"city_name"|"pincode"|"taluka_name"|"district_name"|"state_name"
"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

3. Following copy command will neglect the double quote

2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
ERROR: character with byte sequence 0x9d in encoding "WIN1252" has no equivalent in encoding "UTF8"
CONTEXT: COPY citytemp, line 358646

You need to change the character set on your windows command line before running the script with chcp. Or in postgresql you can:

SET CLIENT_ENCODING TO 'utf8';

Before importing the file.

2ndquadrant.in=# SET CLIENT_ENCODING TO 'utf8';
SET

2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
COPY 679249

 

If  your excel sheet not contain with column name means like ‘cityname’, ‘pincode’..etc means your copy command should be below format

Example : citytemp excel contain below format data’s means COPY Format should be  neglect HEADER

"Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
"Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

COPY FROM for above data format 

COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV;

For Example,

If some column not contains data means you have to use NULL values while using COPY FROM

id,name
1,nijam
2,karthick
   ,nijam
4 ,   
5,

So while copying the values you have to mention value oof null otherwise copy command will throw the error

COPY schema.table FROM 'D:\2ndquadrant\nijam.csv' delimiter ','  CSV header NULL AS 'null';

If you want to insert system date While postgres COPY, for this purpose you have to use now() function
for example data should be below format

date,district
now(),chennai

IS  SEQUENCE UPDATING IMPORTANT ?
You might have noticed that after bulk inserting records using the COPY statement in PostgreSQL the sequence IDs are not getting updated for any further inserts later on, and it would throw duplicate sequence ID errors.
So you would be wondering what makes this COPY statement different that it does not update the sequences,Don’t forget to update the sequence in PostgreSQL after a COPY command.

Use Below anyone of command to update the postgres sequence after bulk insertion or after COPY TO command.

ALTER SEQUENCE ehis.citytemp_seq RESTART WITH 649328;
SELECT setval('ehis.citytemp_seq', 649328);
SELECT setval('ehis.citytemp_seq', 649328);

WHY NEED TO REFRESH POSTGRES MATERIALIZED VIEW ?
–Below could block other connections which are trying to read from the materialized view. this is the preferbale in production environment.

REFRESH MATERIALIZED VIEW CONCURRENTLY ehis.citytemp ;

— Below materilaized view Refreshment both are same , WITH DATA is default

refresh materialized view CONCURRENTLY ehis.citytemp with data;

refresh materialized view CONCURRENTLY ehis.citytemp;

— Refresh the bulk materialized view on particular schema

select 'REFRESH MATERIALIZED VIEW '|| schemaname||'.'||matviewname ||';' as mview from pg_matviews where schemaname ='ehis';

–To finding the materialized view whose name start with ‘city’

select * from pg_matviews where schemaname ='ehis' and matviewname like 'city%' order by matviewname;

–To finding the materialized view whose name with ‘city’

select * from pg_matviews where schemaname ='ehis' and matviewname like '%city%' order by matviewname;

2ndnijam

Leave A Reply

Login with: