• Home
  • Services
    • DBA Support
    • DBA Consultancy Services
    • PostgreSQL Support
    • Website Maintenance
  • Courses

    About Courses

    • List Of Courses
    • Become an Instructor
    Greenplum Database

    Greenplum Database

    $2,000.00 $1,500.00
    Read More
  • Company
    • FAQs
    • About Us
    • Contact
  • Events
  • Portfolio
  • Blogs
    • Blog – RayaFeeL
    • Blog – PostgreSQL Support
    • Blog – PostgreSQL Migration
    • Blog – All DB’s
    • Blog – Linux
    • Blog – Medical Coding
      • Cart

        0

    Have any question?
    (+91)8838953252
    ITsupport@rayafeel.com
    RegisterLogin
    RayaFeeL
    • Home
    • Services
      • DBA Support
      • DBA Consultancy Services
      • PostgreSQL Support
      • Website Maintenance
    • Courses

      About Courses

      • List Of Courses
      • Become an Instructor
      Greenplum Database

      Greenplum Database

      $2,000.00 $1,500.00
      Read More
    • Company
      • FAQs
      • About Us
      • Contact
    • Events
    • Portfolio
    • Blogs
      • Blog – RayaFeeL
      • Blog – PostgreSQL Support
      • Blog – PostgreSQL Migration
      • Blog – All DB’s
      • Blog – Linux
      • Blog – Medical Coding
        • Cart

          0

      Blog

      • Home
      • Blog
      • Blog
      • Postgres Copy Command Tutorial

      Postgres Copy Command Tutorial

      • Posted by 2ndnijam
      • Categories Blog
      • Date February 15, 2019
      • Comments 0 comment

      PostgreSQL COPY data to Text File

      In this post you can learn how to take the backup of particular table by using Postgres COPY command.

      Examples:
      Step1. First list down the table and and describe the table data  which one want to copy from postgres database

      postgres=# select * from batch;
       batch_id |             batch_name              | members 
      ----------+-------------------------------------+---------
              1 | nijam                               |       1
              2 | nijam                               |       5
              3 | nijam                               |       5
              4 | nijam                               |       5
              4 | nijam                               |       8
      (5 rows)

      Step2. then give the copy query which  will copy the table from postgres server to csv file

      postgres=# copy batch to '/opt/PostgreSQL/9.3/data/michel.csv' with delimiter ',' CSV header;
      COPY 5

      step3.  then read the file for copied data whether is correct or not

      [root@p1 ~]# cat /opt/PostgreSQL/9.3/data/michel.csv
      batch_id,batch_name,members
      1,nijam                              ,1
      2,nijam                              ,5
      3,nijam                              ,5
      4,nijam                              ,5
      4,nijam                              ,8

      Example 2.

      second example is same as previous but small difference is here copied data including column name

      postgres=# copy (select * from pg_tables) to '/opt/PostgreSQL/9.3/data/nijam2.csv' 
      with delimiter ',' CSV header; 
      COPY 83
      [root@p1 ~]# cat /opt/PostgreSQL/9.3/data/nijam2.csv
      schemaname,tablename,tableowner,tablespace,hasindexes,hasrules,hastriggers
      pg_catalog,pg_statistic,postgres,,t,f,f
      pg_catalog,pg_type,postgres,,t,f,f
      public,t1,postgres,,f,f,f
      pg_catalog,pg_authid,postgres,pg_global,t,f,f
      pg_catalog,pg_proc,postgres,,t,f,f
      pg_catalog,pg_class,postgres,,t,f,f
      pg_catalog,pg_user_mapping,postgres,,t,f,f
      pg_catalog,pg_attribute,postgres,,t,f,f
      pg_catalog,pg_constraint,postgres,,t,f,f
      pg_catalog,pg_inherits,postgres,,t,f,f
      pg_catalog,pg_index,postgres,,t,f,f
      pg_catalog,pg_operator,postgres,,t,f,f
      pg_catalog,pg_opfamily,postgres,,t,f,f
      pg_catalog,pg_opclass,postgres,,t,f,f
      pg_catalog,pg_am,postgres,,t,f,f
      pg_catalog,pg_amop,postgres,,t,f,f
      pg_catalog,pg_amproc,postgres,,t,f,f
      pg_catalog,pg_language,postgres,,t,f,f
      pg_catalog,pg_largeobject_metadata,postgres,,t,f,f
      pg_catalog,pg_aggregate,postgres,,t,f,f
      pg_catalog,pg_database,postgres,pg_global,t,f,f
      pg_catalog,pg_trigger,postgres,,t,f,f
      pg_catalog,pg_event_trigger,postgres,,t,f,f
      pg_catalog,pg_description,postgres,,t,f,f
      pg_catalog,pg_cast,postgres,,t,f,f
      pg_catalog,pg_enum,postgres,,t,f,f
      pg_catalog,pg_conversion,postgres,,t,f,f
      pg_catalog,pg_depend,postgres,,t,f,f
      pg_catalog,pg_db_role_setting,postgres,pg_global,t,f,f
      pg_catalog,pg_tablespace,postgres,pg_global,t,f,f
      pg_catalog,pg_pltemplate,postgres,pg_global,t,f,f
      pg_catalog,pg_auth_members,postgres,pg_global,t,f,f
      pg_catalog,pg_shdepend,postgres,pg_global,t,f,f
      pg_catalog,pg_shdescription,postgres,pg_global,t,f,f
      pg_catalog,pg_ts_config,postgres,,t,f,f
      pg_catalog,pg_ts_config_map,postgres,,t,f,f
      pg_catalog,pg_ts_dict,postgres,,t,f,f
      pg_catalog,pg_ts_parser,postgres,,t,f,f
      pg_catalog,pg_ts_template,postgres,,t,f,f
      pg_catalog,pg_extension,postgres,,t,f,f
      pg_catalog,pg_foreign_data_wrapper,postgres,,t,f,f
      pg_catalog,pg_foreign_server,postgres,,t,f,f
      pg_catalog,pg_foreign_table,postgres,,t,f,f
      pg_catalog,pg_default_acl,postgres,,t,f,f
      pg_catalog,pg_seclabel,postgres,,t,f,f
      pg_catalog,pg_shseclabel,postgres,pg_global,t,f,f
      pg_catalog,pg_collation,postgres,,t,f,f
      pg_catalog,pg_range,postgres,,t,f,f
      pg_catalog,pg_largeobject,postgres,,t,f,f
      information_schema,sql_implementation_info,postgres,,f,f,f
      information_schema,sql_languages,postgres,,f,f,f
      information_schema,sql_packages,postgres,,f,f,f
      information_schema,sql_parts,postgres,,f,f,f
      public,postgres,postgres,,t,f,f
      information_schema,sql_sizing_profiles,postgres,,f,f,f
      pg_catalog,pg_attrdef,postgres,,t,f,f
      public,primtab,u8,,t,f,f
      public,films,u8,,t,f,f
      public,distributors,u8,,t,f,f
      pg_catalog,pg_namespace,postgres,,t,f,f
      pg_catalog,pg_rewrite,postgres,,t,f,f
      public,k,u8,,t,f,f
      public,distributorc,u8,,f,f,f
      public,distributorsct,u8,,f,f,f
      public,postgres1,postgres,,t,f,f
      public,postgres2,postgres,,t,f,f
      public,postgres3,postgres,,t,f,f
      public,postgres4,postgres,,t,f,f
      public,postgres_5,postgres,,t,f,f
      public,students,postgres,,t,f,f
      public,india,postgres,,t,f,f
      information_schema,sql_features,postgres,,f,f,f
      information_schema,sql_sizing,postgres,,f,f,f
      public,batch,postgres,,t,f,f
      public,cellphones,postgres,,t,f,f
      public,mobiles,postgres,,t,f,f
      public,location,postgres,,t,f,f
      public,dept,postgres,,f,f,f
      public,products1,postgres,,f,f,f
      public,products2,postgres,,f,f,f
      public,t7,postgres,,f,f,f
      public,village,postgres,,f,f,f
      public,emp_d,postgres,,f,f,f

      PostgreSQL COPY data From Text File

      Here i’m Explained about How to insert the data into postgres database from text file.

      Examples:
      insert the data into the database from text file 

      Step1. create the text file  and insert some data into text file.
      [root@p1 ~]# vi /opt/PostgreSQL/9.3/data/new.txt
      [root@p1 ~]# cat /opt/PostgreSQL/9.3/data/new.txt
      1,akash
      2,varun
      3,makash
      4,nijam
      5,benz

      step2. create a table as per text file structure here i created the table with to column  because in text file having 2 column one is number which you can consider id with integer data type second is name which is name with text data type

      postgres=# create table dept(id integer,name text);
      CREATE TABLE

      step3. Now you can issue the query  which will copy the data from text file to database

      postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
      ERROR:  could not open file "/opt/PostgreSQL/9.3/data/new.txt" for writing: Permission denied

      Because new.txt file is root user file so you have to change the file ownership to postgres user
      Solution:

      [root@p1 ~]# chown postgres:postgres /opt/PostgreSQL/9.3/data/new.txt
      [root@p1 ~]# cd /opt/PostgreSQL/9.3/data/
      
      [root@p1 data]# ls -lrt new.txt 
      -rw-r--r-- 1 postgres postgres 43 Apr  3 16:44 new.txt
      
      postgres=# copy dept from '/opt/PostgreSQL/9.3/data/new.txt' with delimiter ',';
      COPY 5
      postgres=# select * from dept;
       id |  name  
      ----+--------
        1 | akash
        2 | varun
        3 | makash
        4 | nijam
        5 | benz
      (5 rows)

      Example 2. this is same as previous  example but only thing is  here added only one extra column which is date

      — check the contents of nijam.csv file which will be helpful to creating table structure
      [root@p1 data]# vi nijam.csv
      
      [root@p1 data]# cat nijam.csv 
      1,      mala,20-SEP-90 00:00:00
      2,      tarun,10-OCT-08 00:00:00
      3,      lara,12-JAN-09 00:00:00
      4,      sara,08-MAY-01 00:00:00
      5,      guttu,10-MAY-09 00:00:00
      
      [root@p1 data]# ls -lrt nijam.csv 
      -rw-r--r-- 1 root root 174 Apr  3 17:22 nijam.csv
      
      -- check the file permission file ownership permission should be postgres user
      [root@p1data]# pwd
      /opt/PostgreSQL/9.3/data/
      
      [root@p1 data]# chown postgres:postgres /opt/PostgreSQL/9.3/data/nijam.csv 
      [root@p1 data]# ls -lrt nijam.csv 
      -rw-r--r-- 1 postgres postgres 174 Apr  3 17:22 nijam.csv

      — then create the table structure as per data structures then import the data into postgresql server

      postgres=# create table emp_d(id integer,name text, dob date);
      CREATE TABLE
      
      postgres=# copy emp_d from '/opt/PostgreSQL/9.3/data/nijam.csv' with delimiter ',';
      COPY 5
      postgres=# select * from emp_d;
       id |    name     |    dob     
      ----+-------------+------------
        1 |       mala  | 1990-09-20
        2 |       tarun | 2008-10-10
        3 |       lara  | 2009-01-12
        4 |       sara  | 2001-05-08
        5 |       guttu | 2009-05-10
      (5 rows)

      More Example postgres COPY command

      Below  Examples are worked on postgres windows server so you can  use ‘COPY ‘ instead of using ‘ \COPY ‘.

      While Importing CSV files with text fields that contain double quotes using postgres COPY .

      By using below copy command you can restore the postgres data into the target table if data has been double quotes as well.

      1. connecting windows server

      D:
      cd D:\PostgresPlus\9.4AS\bin
      psql -U enterprisedb -d Health_DC

      2. Sample Data’s :

      "city_name"|"pincode"|"taluka_name"|"district_name"|"state_name"
      "Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

      3. Following copy command will neglect the double quote

      2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
      ERROR: character with byte sequence 0x9d in encoding "WIN1252" has no equivalent in encoding "UTF8"
      CONTEXT: COPY citytemp, line 358646

      You need to change the character set on your windows command line before running the script with chcp. Or in postgresql you can:

      SET CLIENT_ENCODING TO 'utf8';

      Before importing the file.

      2ndquadrant.in=# SET CLIENT_ENCODING TO 'utf8';
      SET
      
      2ndquadrant.in=# COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV HEADER;
      COPY 679249

       

      If  your excel sheet not contain with column name means like ‘cityname’, ‘pincode’..etc means your copy command should be below format

      Example : citytemp excel contain below format data’s means COPY Format should be  neglect HEADER

      "Arong"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Sawai"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Chuckchucha"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Kinyuka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Big Lapati"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"
      "Perka"|"744301"|"Car Nicobar"|"Nicobar"|"Andaman and Nicobar Islands"

      COPY FROM for above data format 

      COPY ehis.citytemp FROM 'D:\2ndq/nijamutheen.csv' DELIMITER '|' CSV;

      For Example,

      If some column not contains data means you have to use NULL values while using COPY FROM

      id,name
      1,nijam
      2,karthick
         ,nijam
      4 ,   
      5,

      So while copying the values you have to mention value oof null otherwise copy command will throw the error

      COPY schema.table FROM 'D:\2ndquadrant\nijam.csv' delimiter ','  CSV header NULL AS 'null';

      If you want to insert system date While postgres COPY, for this purpose you have to use now() function
      for example data should be below format

      date,district
      now(),chennai

      IS  SEQUENCE UPDATING IMPORTANT ?
      You might have noticed that after bulk inserting records using the COPY statement in PostgreSQL the sequence IDs are not getting updated for any further inserts later on, and it would throw duplicate sequence ID errors.
      So you would be wondering what makes this COPY statement different that it does not update the sequences,Don’t forget to update the sequence in PostgreSQL after a COPY command.

      Use Below anyone of command to update the postgres sequence after bulk insertion or after COPY TO command.

      ALTER SEQUENCE ehis.citytemp_seq RESTART WITH 649328;
      SELECT setval('ehis.citytemp_seq', 649328);
      SELECT setval('ehis.citytemp_seq', 649328);

      WHY NEED TO REFRESH POSTGRES MATERIALIZED VIEW ?
      –Below could block other connections which are trying to read from the materialized view. this is the preferbale in production environment.

      REFRESH MATERIALIZED VIEW CONCURRENTLY ehis.citytemp ;

      — Below materilaized view Refreshment both are same , WITH DATA is default

      refresh materialized view CONCURRENTLY ehis.citytemp with data;
      
      refresh materialized view CONCURRENTLY ehis.citytemp;

      — Refresh the bulk materialized view on particular schema

      select 'REFRESH MATERIALIZED VIEW '|| schemaname||'.'||matviewname ||';' as mview from pg_matviews where schemaname ='ehis';

      –To finding the materialized view whose name start with ‘city’

      select * from pg_matviews where schemaname ='ehis' and matviewname like 'city%' order by matviewname;

      –To finding the materialized view whose name with ‘city’

      select * from pg_matviews where schemaname ='ehis' and matviewname like '%city%' order by matviewname;

      • Share:
      2ndnijam

      Previous post

      Postgres Database Performance Tuning
      February 15, 2019

      Next post

      Postgres transaction isolation issue ?
      February 16, 2019

      Leave A Reply Cancel reply

      You must be logged in to post a comment.

      Login with:

      Login with Google Login with Twitter Login with LinkedIn Login with Microsoft


      Search

      ADVERTISEMENT

      Latest Posts

      PostgreSQL Patching version 9, 10,11
      10Oct2019
      Tools for PostgreSQL
      16Sep2019
      Postgres user creation and restrict DDL & database access
      13Sep2019
      PostgreSQL SSL Setup
      07Sep2019
      How to DELETE current XLOG / WAL LOG in postgresql database ?
      19Aug2019

      Latest Courses

      PostgreSQL Database

      PostgreSQL Database

      $600.00 $500.00
      Greenplum Database

      Greenplum Database

      $2,000.00 $1,500.00

      Preview Course

      Free

      Recent Forum Topics

      • thought behind whiteboard activity
      • Are you going to take your first ste
      • How to start working on an application?
      • please let me know pre requirements to increase work_mem
      • how to copy some data in one table to another table in postgres

      2ndquadrant.in

      (+91) 8838953252

      ITsupport@rayafeel.com

      Company

      • About Us
      • Contact
      • Our Team
      • Blog

      COURSES

      • List Of Course
      • Become An Instructor
      • Events
      • Postgres Support Blog

      Support

      • DBA Support
      • Consultancy Services
      • Postgres Migration Blogs
      • Forum

      Recommend

      • Groups
      • Login
      • FAQs
      • SignUp

      IT Services by rayafeel.com. Powered by Rayafeel Technologies Pvt Ltd.

      • Privacy
      • Terms

      Become An Instructor?

      Join thousand of instructors and earn money hassle free!

      Get Started Now

      Login with:

      Login with Google Login with Twitter Login with LinkedIn Login with Microsoft

      Login with your site account

      Lost your password?

      Not a member yet? Register now

      Register a new account

      Are you a member? Login now