database application help.

This is a discussion on database application help. within the Linux Programming forums, part of the Platform Specific Boards category; I don't know if this is the right board for this question, but I'm coding this application on a linux ...

  1. #1
    Billy Baroo
    Guest

    database application help.

    I don't know if this is the right board for this question, but I'm coding this application on a linux platform.

    I'm basically new programming, and I'm teaching myself. I have a basic question, which is independent of language, and I know there are some smart people here who could help me out.

    I'm building an application that will retrieve two large files from a remote FTP server, parse the files, and update a database with the new data from the files. I'm coding this in Python. My question is this: Which of the two solutions would be the way to go and why? (also, if you know of a better way please let me know).

    Solution 1 (memory issues?):
    1. Establish a connection with the server.
    2. Read the contents of the files into two large data structures
    3. Close connection with FTP server.
    4. Open connection with SQL database.
    5. Iterate through data structures and update database records.
    6. Close connection to Database

    Solution 2 (device read and write issues?):
    1. Establish connection with FTP server
    2. Establish connection with SQL server
    3. While there are lines in the files, retrieve and parse data line by line from FTP server update corresponding record in the database.
    4. Close connections to both servers.

    Since i'm coding this is python, you can guess that speed isn't really an issue (it'll will run very early in the morning), but resource consumption is important. I just want to know the least retarded way to handle the problem.

    I'm just wondering how you handle large amounts of data, and I don't really understand what bottle necks would occur with each solution. If anyone has any ideas, please let me know.

  2. #2
    Registered User
    Join Date
    Jul 2002
    Posts
    161
    I would go for Solution 1, download the two files first then update. Consider, what if the ftp server resets in the middle of updating your database? Method 2 could leave things in a terrible mess. Probably you could just start over but I try to program failsafe, so nothing possibly can go wrong, go wrong, go wrong.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Creating a database
    By Shamino in forum Game Programming
    Replies: 19
    Last Post: 06-10-2007, 01:09 PM
  2. Best choise for a local application database
    By gogic in forum C# Programming
    Replies: 2
    Last Post: 04-04-2006, 07:08 AM
  3. Using C++ for GUI with Database Application
    By bugmenot in forum Windows Programming
    Replies: 7
    Last Post: 12-14-2005, 07:24 PM
  4. Database Application VC++ or vb6..?
    By gardenair in forum C Programming
    Replies: 0
    Last Post: 04-13-2003, 10:06 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21