Guidance regarding 5 gb svn dump import

Can someone provide input as to expectations for a subversion import from a dump file that is nearly 5 gb? Has anyone performed an import with a svn dump that is this large? Any problems or things be aware of?

I'm sure the SCM team or others will chime in here soon but one thing you may want to be aware of is that if you're using the DB2 Express Edition bundled with RTC, there may be issues. With RTC 1.0.1.1 we upgraded to a new version of DB2 that fixed intermittent issues that would sometimes occur when importing from CVS or SVN. You can read more about it here:

https://jazz.net/downloads/rational-team-concert/releases/1.0.1.1?p=upgrading

Look under Upgrading to DB2 Express Edition 9.5 Fixpack 3a where you'll also find links to the DB2 installers. More information about this is in work item 70878.

We have done imports of dump files up that were > 6 GB.
SVN Imports are resumable as well, so if you lose connectivity half way
through, you can restart without having to redo what you have already
done.

Hope this helps,
JohnC
SCM Server

tdunnigan wrote:

Can someone provide input as to expectations for a subversion import
from a dump file that is nearly 5 gb? Has anyone performed an import
with a svn dump that is this large? Any problems or things be aware
of?

We are trying to import a subversion dump file of size 5gb locally on the RTC server, but import process fails with following error after processing all the revisions (subversion dump contains approx 70000 revisions)

An I/O error occurred while trying to create change set archive d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content tablespace pages" to 100000000 instead of the default 2500000 at com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without any success.

please help me resolve this

Thanks,
Aruljothi

aruljothi wrote:

We are trying to import a subversion dump file of size 5gb locally on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000 at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

Hello Aruljothi,

Please file a bug on jazz.net against Source Control, we will
troubleshoot the problem there ASAP. Please attach the full stack trace
if you have it from the Eclipse error log.

Thanks
John Camelon
SCM Server

Aruljothi,

How much free disk space do you have on your D drive? The importer
writes a slightly compressed archive of the dump file to aid in
importing so you would need 5-10 gig of free space to write the archive.
Also, please check the error log (you can use the Error Log view), to
see if there is a stack trace provided for the I/O exception.

Michael

aruljothi wrote:

We are trying to import a subversion dump file of size 5gb locally on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000 at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

We have more than enough space
This is the log file from the effort Arul and I have been working on:


eclipse.buildId=I20080617-2000
java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223-20080315 (JIT enabled)
J9VM - 20080314_17962_lHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Framework arguments: -product com.ibm.team.concert.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product com.ibm.team.concert.product


Error
Wed Mar 25 10:22:00 EDT 2009
An I/O error occurred while trying to create change set archive C:\svndump\archive.tar

com.ibm.team.repository.common.TeamRepositoryException: An I/O error occurred while trying to create change set archive C:\svndump\archive.tar
at com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:54)
at com.ibm.team.scm.client.importz.internal.CreateArchiveOperation.run(CreateArchiveOperation.java:32)
at com.ibm.team.scm.client.importz.ui.ImportOperation.createArchive(ImportOperation.java:122)
at com.ibm.team.scm.client.importz.ui.ImportOperation.filesystemRun(ImportOperation.java:93)
at com.ibm.team.filesystem.ui.operations.FileSystemUIOperation.repositoryRun(FileSystemUIOperation.java:43)
at com.ibm.team.repository.rcp.ui.operations.RepositoryOperation.run(RepositoryOperation.java:42)
at com.ibm.team.repository.rcp.ui.operations.JobRunner.run(JobRunner.java:205)
at com.ibm.team.repository.rcp.ui.operations.JobRunner$1.run(JobRunner.java:62)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.IOException: Expected primitive property but found "/*--------------------------------------------------------------------*
"
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.signalError(SVNDumpReader.java:306)
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readPrimitiveProperties(SVNDumpReader.java:209)
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readProperties(SVNDumpReader.java:173)
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.readNode(SVNDumpProcessor.java:93)
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:80)
at com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:57)
at com.ibm.team.scm.client.importz.svn.internal.SVNDump2ArchiverConverter.createArchive(SVNDump2ArchiverConverter.java:86)
at com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:49)
... 8 more

Aruljothi,

How much free disk space do you have on your D drive? The importer
writes a slightly compressed archive of the dump file to aid in
importing so you would need 5-10 gig of free space to write the archive.
Also, please check the error log (you can use the Error Log view), to
see if there is a stack trace provided for the I/O exception.

Michael

aruljothi wrote:

We are trying to import a subversion dump file of size 5gb locally on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000 at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

It looks like the format of the dump file is not what was expected. What
Subversion version did you use to generate the dump file? What command
line parameters did you use to generate the dump file? We've never seen
an error like this before so there may be something different about your
server version or the command you used to generate the dump file that is
contributing to the error. Could you please log a defect against Source
Control in the Rational Team Concert project area of Jazz.net with this
information?

Thanks,
Michael

tdunnigan wrote:

We have more than enough space
This is the log file from the effort Arul and I have been working
on:


eclipse.buildId=I20080617-2000
java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32
j9vmwi3223-20080315 (JIT enabled)
J9VM - 20080314_17962_lHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Framework arguments: -product com.ibm.team.concert.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product
com.ibm.team.concert.product


Error
Wed Mar 25 10:22:00 EDT 2009
An I/O error occurred while trying to create change set archive
C:\svndump\archive.tar

com.ibm.team.repository.common.TeamRepositoryException: An I/O error
occurred while trying to create change set archive
C:\svndump\archive.tar
at
com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:54)
at
com.ibm.team.scm.client.importz.internal.CreateArchiveOperation.run(CreateArchiveOperation.java:32)
at
com.ibm.team.scm.client.importz.ui.ImportOperation.createArchive(ImportOperation.java:122)
at
com.ibm.team.scm.client.importz.ui.ImportOperation.filesystemRun(ImportOperation.java:93)
at
com.ibm.team.filesystem.ui.operations.FileSystemUIOperation.repositoryRun(FileSystemUIOperation.java:43)
at
com.ibm.team.repository.rcp.ui.operations.RepositoryOperation.run(RepositoryOperation.java:42)
at
com.ibm.team.repository.rcp.ui.operations.JobRunner.run(JobRunner.java:205)
at
com.ibm.team.repository.rcp.ui.operations.JobRunner$1.run(JobRunner.java:62)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.IOException: Expected primitive property but found
"/*--------------------------------------------------------------------*
"
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.signalError(SVNDumpReader.java:306)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readPrimitiveProperties(SVNDumpReader.java:209)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readProperties(SVNDumpReader.java:173)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.readNode(SVNDumpProcessor.java:93)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:80)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:57)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDump2ArchiverConverter.createArchive(SVNDump2ArchiverConverter.java:86)
at
com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:49)
.. 8 more



Michael Valentawrote:
Aruljothi,
How much free disk space do you have on your D drive? The importer
writes a slightly compressed archive of the dump file to aid in
importing so you would need 5-10 gig of free space to write the
archive.
Also, please check the error log (you can use the Error Log view),
to
see if there is a stack trace provided for the I/O exception.

Michael

aruljothi wrote:
We are trying to import a subversion dump file of size 5gb locally
on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000
at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

The dump file was created using the polarian pvcs2svn tooling. I fear that may be culprit. Prior to logging a defect, we are importing into a subversion server and recreating the dump file from there.

It looks like the format of the dump file is not what was expected. What
Subversion version did you use to generate the dump file? What command
line parameters did you use to generate the dump file? We've never seen
an error like this before so there may be something different about your
server version or the command you used to generate the dump file that is
contributing to the error. Could you please log a defect against Source
Control in the Rational Team Concert project area of Jazz.net with this
information?

Thanks,
Michael

tdunnigan wrote:

We have more than enough space
This is the log file from the effort Arul and I have been working
on:


eclipse.buildId=I20080617-2000
java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32
j9vmwi3223-20080315 (JIT enabled)
J9VM - 20080314_17962_lHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Framework arguments: -product com.ibm.team.concert.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product
com.ibm.team.concert.product


Error
Wed Mar 25 10:22:00 EDT 2009
An I/O error occurred while trying to create change set archive
C:\svndump\archive.tar

com.ibm.team.repository.common.TeamRepositoryException: An I/O error
occurred while trying to create change set archive
C:\svndump\archive.tar
at
com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:54)
at
com.ibm.team.scm.client.importz.internal.CreateArchiveOperation.run(CreateArchiveOperation.java:32)
at
com.ibm.team.scm.client.importz.ui.ImportOperation.createArchive(ImportOperation.java:122)
at
com.ibm.team.scm.client.importz.ui.ImportOperation.filesystemRun(ImportOperation.java:93)
at
com.ibm.team.filesystem.ui.operations.FileSystemUIOperation.repositoryRun(FileSystemUIOperation.java:43)
at
com.ibm.team.repository.rcp.ui.operations.RepositoryOperation.run(RepositoryOperation.java:42)
at
com.ibm.team.repository.rcp.ui.operations.JobRunner.run(JobRunner.java:205)
at
com.ibm.team.repository.rcp.ui.operations.JobRunner$1.run(JobRunner.java:62)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.IOException: Expected primitive property but found
"/*--------------------------------------------------------------------*
"
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.signalError(SVNDumpReader.java:306)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readPrimitiveProperties(SVNDumpReader.java:209)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readProperties(SVNDumpReader.java:173)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.readNode(SVNDumpProcessor.java:93)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:80)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:57)
at
com.ibm.team.scm.client.importz.svn.internal.SVNDump2ArchiverConverter.createArchive(SVNDump2ArchiverConverter.java:86)
at
com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:49)
.. 8 more



Michael Valentawrote:
Aruljothi,
How much free disk space do you have on your D drive? The importer
writes a slightly compressed archive of the dump file to aid in
importing so you would need 5-10 gig of free space to write the
archive.
Also, please check the error log (you can use the Error Log view),
to
see if there is a stack trace provided for the I/O exception.

Michael

aruljothi wrote:
We are trying to import a subversion dump file of size 5gb locally
on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000
at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

Yes, we have had problem reports from other users that tried to use the
pvcs2svn tool. It doesn't generate a well formed SVN dump file and it
apparently doesn't recreate the branch and tag history properly (i.e.
branches and tags are not linked to the original copy that created them
(or, at least it didn't in the one case I know about).

Michael

tdunnigan wrote:

The dump file was created using the polarian pvcs2svn tooling. I fear
that may be culprit. Prior to logging a defect, we are importing
into a subversion server and recreating the dump file from there.

Michael Valentawrote:
It looks like the format of the dump file is not what was expected.
What
Subversion version did you use to generate the dump file? What
command
line parameters did you use to generate the dump file? We've never
seen
an error like this before so there may be something different about
your
server version or the command you used to generate the dump file
that is
contributing to the error. Could you please log a defect against
Source
Control in the Rational Team Concert project area of Jazz.net with
this
information?

Thanks,
Michael

tdunnigan wrote:
We have more than enough space
This is the log file from the effort Arul and I have been working
on:


eclipse.buildId=I20080617-2000
java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32
j9vmwi3223-20080315 (JIT enabled)
J9VM - 20080314_17962_lHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Framework arguments: -product com.ibm.team.concert.product
Command-line arguments: -os win32 -ws win32 -arch x86 -product
com.ibm.team.concert.product


Error
Wed Mar 25 10:22:00 EDT 2009
An I/O error occurred while trying to create change set archive
C:\svndump\archive.tar

com.ibm.team.repository.common.TeamRepositoryException: An I/O
error
occurred while trying to create change set archive
C:\svndump\archive.tar
at

com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:54)
at

com.ibm.team.scm.client.importz.internal.CreateArchiveOperation.run(CreateArchiveOperation.java:32)
at

com.ibm.team.scm.client.importz.ui.ImportOperation.createArchive(ImportOperation.java:122)
at

com.ibm.team.scm.client.importz.ui.ImportOperation.filesystemRun(ImportOperation.java:93)
at

com.ibm.team.filesystem.ui.operations.FileSystemUIOperation.repositoryRun(FileSystemUIOperation.java:43)
at

com.ibm.team.repository.rcp.ui.operations.RepositoryOperation.run(RepositoryOperation.java:42)
at

com.ibm.team.repository.rcp.ui.operations.JobRunner.run(JobRunner.java:205)
at

com.ibm.team.repository.rcp.ui.operations.JobRunner$1.run(JobRunner.java:62)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.IOException: Expected primitive property but
found
"/*--------------------------------------------------------------------*
"
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.signalError(SVNDumpReader.java:306)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readPrimitiveProperties(SVNDumpReader.java:209)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpReader.readProperties(SVNDumpReader.java:173)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.readNode(SVNDumpProcessor.java:93)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:80)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDumpProcessor.read(SVNDumpProcessor.java:57)
at

com.ibm.team.scm.client.importz.svn.internal.SVNDump2ArchiverConverter.createArchive(SVNDump2ArchiverConverter.java:86)
at

com.ibm.team.scm.client.importz.svn.internal.SVNImporter.createChangeSetArchive(SVNImporter.java:49)
.. 8 more



Michael Valentawrote:
Aruljothi,
How much free disk space do you have on your D drive? The importer
writes a slightly compressed archive of the dump file to aid in
importing so you would need 5-10 gig of free space to write the
archive.
Also, please check the error log (you can use the Error Log view),
to
see if there is a stack trace provided for the I/O exception.

Michael

aruljothi wrote:
We are trying to import a subversion dump file of size 5gb locally
on
the RTC server, but import process fails with following error after
processing all the revisions (subversion dump contains approx 70000
revisions)

An I/O error occurred while trying to create change set archive
d:\temp\1\importArchive26004.tmp

We have also tried setting the property "Number of content
tablespace pages" to 100000000 instead of the default 2500000
at
com.ibm.team.repository.jdbcdriver.internal.db2.Db2Provider without
any success.

please help me resolve this

Thanks,
Aruljothi

I have a similar error, but created mine using svn dump via svn 1.6.11 from CollabNet on Linux, I moved the dump to windows, and am trying the import from there.