It's all about the answers!

Ask a question

Fetching to an existing source tree


Rick Chapman (7111312) | asked Jan 07 '11, 5:57 p.m.
To reduce fetch times we're now saving a copy of the last fetched source tree for our builds and then using that to seed the next build's source tree before doing the fetch for it. I have a few questions about this approach (note: we are using the ant toolkit for all our rtc interactions and are running rtc v2 at this time):
1. Is there a way to see the list of files that end up being fetched?
2. What happens when a file rename or file delete was delivered? Does the fetch know to remove the old file?
3. Regarding question 2, if the old file remains, is there a way to determine that it should be removed so our build script can take care of that after the fetch finishes?
4. What criteria does the fetch use to decide if a file should be downloaded or not? Does it compare file size, date or checksum? Is that customizable?
Thanks for any clarification you give me on this.

13 answers



permanent link
Michael Valenta (3.7k3) | answered Jan 10 '11, 11:00 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
When you say you are saving a copy of the source tree, do you mean that you are saving a copy of the sandbox (i.e. the folder that you load the files into) or just the folder and files that were loaded. I ask because the sandbox contains some metadata stored in the .jazz5 folder that contaisn information on the state of the files that were loaded. If that is present, RTC would use that data to determine what files need to be loaded, including files that are to be deleted or moved. However, if the metadata is not present, i suspect the current content would be deleted and new content loaded (although I'm not 100% sure on that). Also, I do not believe that there is a way to determine what new files are loaded from within the ant task.

Hope this helps,

permanent link
Jeff Care (1.0k3833) | answered Jan 10 '11, 4:51 p.m.
I'm also very interested in getting data about new/changed/removed files for an incremental extract. Having this information would be very valuable for doing incremental builds.

permanent link
David Lafreniere (4.8k7) | answered Jan 11 '11, 12:08 p.m.
FORUM MODERATOR / JAZZ DEVELOPER
I'm also very interested in getting data about new/changed/removed files for an incremental extract. Having this information would be very valuable for doing incremental builds.


Hi Jeff, one way to do this is to use the "generateChangeLog" Ant task that is in our Buildsystem toolkit. The generateChangeLog task compares two snapshots (note: snapshots can be created with every build using the Jazz Source Control pre-build participant) to generate a list of change sets, added components, and removed components. You can then use the logPublisher task to publish the change log file to a specific build result or do what ever you want with it.

See this URL for additional details:
http://publib.boulder.ibm.com/infocenter/clmhelp/v3r0/index.jsp?topic=/com.ibm.team.build.doc/topics/t_build_overview.html

permanent link
Rick Chapman (7111312) | answered Feb 01 '11, 6:12 p.m.
I'm still really hoping someone from the RTC team will respond to the 4 questions listed in my original post regarding fetching to a directory that contains the results of a previous fetch. To answer a question that was raised, yes, all the .jazz5 files are still present from the previous fetch when the second fetch is done. Thanks in advance for any answers you can give me.

permanent link
ASHEESH PRAJAPATI (16) | answered Feb 02 '11, 7:15 p.m.
Hi All,

I am wondering if there is a way in RTC to identify the incremental changes done since last build and download only those changes to file system?

I am working with the packaged application where I just need the delta changes to perform the incremental compile and any easier way to achieve this will make the things easy.

Currently I am executing the generatechangelog and read the output file to identify changes and then copy those to separate folder for further processing.

Hoping someone can suggest any better way of doing that using native RTC commands.

Thank you
Asheesh

permanent link
Michael Valenta (3.7k3) | answered Feb 04 '11, 10:27 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
I'll try and answer your four questions with the assumption that you are fetching into a sandbox that has the metadata from the previous fetch.

1. You may be able to get the list of change files using generateChangeLog feature that was mentioned by David above. I haven't used it myself so I can;t say for sure.

2. File renames and deletes will be properly handled since the metadata of the sandbox keeps track of the state of the items that were previously loaded.

3. See point 2

4. Files will be download if the file file has a change in the repository since it was last loaded or if there is a local change (determined by comparing timestamp, file size and content hash). This is not customizable.

permanent link
Rick Chapman (7111312) | answered Jun 09 '11, 3:54 p.m.
> File renames and deletes will be properly handled since the metadata
> of the sandbox keeps track of the state of the items that were previously
> loaded.

This doesn't seem to be working for me. The customer delivered some file deletes and, from the GUI Show->Repository Files I confirmed that they are no longer there. When my build runs, it lays down the previous build's source tree, which includes those deleted files since they existed at the time of the last build. Then an accept and a fetch is run and, when the fetch is done, I see those files are still there. You said that they would be deleted assuming the build is fetching into a sandbox that has the metadata from the previous fetch. I'm still a little fuzzy on what that means. When I seed the new build tree by laying down the last build's saved source tree, it includes the .jazz5 tree and everything in the tree as it was just after the last build's fetch. Is the .jazz5 tree the "metadata of the sandbox"? Or is there something else that needs to be saved so that the fetch knows to remove deleted files when it's fetching into a tree that's already full of source code from a previous fetch? I'd appreciate any help you can give me to understand what I'm doing wrong and how to get this working.

permanent link
Rick Chapman (7111312) | answered Jun 16 '11, 3:41 p.m.
Repeating last entry hoping someone will have mercy on me and help...

> File renames and deletes will be properly handled since the metadata
> of the sandbox keeps track of the state of the items that were previously
> loaded.

This doesn't seem to be working for me. The customer delivered some file deletes and, from the GUI Show->Repository Files I confirmed that they are no longer there. When the build runs, it lays down the previous build's source tree, which includes those deleted files since they existed at the time of the last build. Then an accept and a fetch is run and, when the fetch is done, I see those files are still there. You said that they would be deleted assuming the build is fetching into a sandbox that has the metadata from the previous fetch. I'm still a little fuzzy on what that means. When I seed the new build tree by laying down the last build's saved source tree, it includes the .jazz5 tree and everything in the tree as it was just after the last build's fetch. Is the .jazz5 tree the "metadata of the sandbox"? Or is there something else that needs to be saved so that the fetch knows to remove deleted files when it's fetching into a tree that's already full of source code from a previous fetch? I'd appreciate any help you can give me to understand what I'm doing wrong and how to get this working.

permanent link
Geoffrey Clemm (30.1k33035) | answered Jun 16 '11, 9:14 p.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
What specific commands are you running?
When you say "lays down the previous build's source tree", I assume it
is doing so by restoring some copy you made and stored somewhere. By
"accept and fetch", are you using the RTC scm "load" operation, or the
"accept" operation, or some combination of the two?

A question for the command line experts:
Assuming the above is true, and a copy (including the .jazz5 metadata)
is being manually loaded somewhere on disk, what does he have to do to
let Jazz know that this tree exists? A "load" operation? I'm
assuming that an "accept" operation by itself would have no effect on
the files on disk, because the scm client doesn't know that this tree
exists, since it was created without the scm system knowing about it.

Cheers,
Geoff

On 6/16/2011 3:53 PM, chtbuild wrote:
Repeating last entry hoping someone will have mercy on me and help...

File renames and deletes will be properly handled since the
metadata
of the sandbox keeps track of the state of the items that were
previously
loaded.

This doesn't seem to be working for me. The customer delivered some
file deletes and, from the GUI Show->Repository Files I confirmed
that they are no longer there. When the build runs, it lays down the
previous build's source tree, which includes those deleted files since
they existed at the time of the last build. Then an accept and a
fetch is run and, when the fetch is done, I see those files are still
there. You said that they would be deleted assuming the build is
fetching into a sandbox that has the metadata from the previous
fetch. I'm still a little fuzzy on what that means. When I seed the
new build tree by laying down the last build's saved source tree, it
includes the .jazz5 tree and everything in the tree as it was just
after the last build's fetch. Is the .jazz5 tree the "metadata
of the sandbox"? Or is there something else that needs to be
saved so that the fetch knows to remove deleted files when it's
fetching into a tree that's already full of source code from a
previous fetch? I'd appreciate any help you can give me to
understand what I'm doing wrong and how to get this working.

permanent link
Rick Chapman (7111312) | answered Jun 20 '11, 11:23 a.m.
Geoff, thanks for the reply.

> When you say "lays down the previous build's source tree", I assume it
> is doing so by restoring some copy you made and stored somewhere.
Correct.

> By "accept and fetch", are you using the RTC scm "load" operation, or
> the "accept" operation, or some combination of the two?
I am using the ANT toolkit teamAccept and teamFetch tasks. I only mention the accept so you can understand the context. I assume the accept only updates the workspace with newly delivered items. The fetch then downloads the workspace content and that is the operation in question. Perhaps the command line "load" action behaves differently than the ANT teamFetch with regards to the use of the metadata? If that is the case, I could try replacing my teamFetch with the load command but I'd like to know if that is the solution before I start rewriting things.

Thanks again for your help.

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.