annotate src/breadcrumbs/docs/Tutorial-BreadCrumbs.md @ 3:b4cf8c75305b draft default tip

Pointing to the right root directory
author george-weingart
date Tue, 30 Aug 2016 13:10:34 -0400
parents 2f4f6f08c8c4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
1 # BreadCrumbs Tutorial #
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
2
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
3 This is a brief tutorial to get you acquainted with the scripts provided in breadcrumbs. This tutorial is oragnized by script and task. Examples are given using files in the demo_input folder which is included in the BreadCrumbs package. Each of these commands should work from the command line in the breadcrumbs directory.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
4
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
5 Please note all of the following calls expect you to be in the breadcrumbs directory and to have both the ./breadcrumbs/src and ./breadcrumbs/scripts in your path and or python path.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
6
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
7 Enjoy and happy research!
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
8
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
9 ## Contents: ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
10 1. scriptPCoA
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
11 2. scriptManipulateTable.py
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
12 I. Manipulating the measurements
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
13 II. Filtering
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
14 III. Filtering with knowledge of feature hierarchical relationship
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
15 IV. Manipulate samples by metadata
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
16 V. Manipulate the feature names
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
17 3. scriptPlotFeature.py
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
18 4. scriptBiplotTSV.R
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
19 5. scriptConvertBetweenBIOMAndPCL.py
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
20
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
21 ## scriptPCoA.py ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
22 This script allows one to plot a PCoA of an abundance table. In the plot each sample is one marker. The marker shape and color is determined by a metadata (of your choice). The distances between each sample is determined by a specific beta-diversity distance metric. By default Bray-curtis distance is used. This can be changed as needed. You will notice for every call you must give it the sample id (-i) and the last metadata which should be the row before your first data (-l). This helps the scripts understand what is a data measurement and what is a metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
23
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
24 A. How do I make a PCoA of an abundance table, painting (coloring) it by a specific metadata?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
25
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
26 > scripts/scriptPcoa.py -i TID -l STSite -p STSite demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
27
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
28 B. How do I make a series of PCoAs of an abundance table, one PCoA for every metadata?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
29
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
30 If nothing is specified with -p then all metadata are painted. Note there are a max of 9 shapes to use, a metadata will be skipped if it has more than 9 levels (specific values which can be used many times). Don't worry, the script will let you know if this happens and will just skip to the next metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
31
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
32 > scripts/scriptPcoa.py -i TID -l STSite demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
33
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
34 C. How do I use a different beta-diversity distance metric instead of Bray-curtis distance?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
35 The following metrics can be choosen: braycurtis, canberra, chebyshev, cityblock, correlation, cosine, euclidean, hamming, sqeuclidean, unifrac_unweighted, unifrac_weighted
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
36
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
37 > scripts/scriptPcoa.py -i TID -l STSite -m sqeuclidean demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
38
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
39 D. How do I get the coordinates of the points in the PCoA plot? Use -C and give a file path to which to write.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
40
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
41 > scripts/scriptPcoa.py -i TID -l STSite -C coordinates.txt demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
42
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
43 E. How do I get the distance matrix represented by the PCoA plot? Use -D and give a file path to which to write.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
44
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
45 > scripts/scriptPcoa.py -i TID -l STSite -D distances.txt demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
46
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
47 F. How do I make a PCoA using unifrac type metrics.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
48
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
49 > scripts/scriptPcoa.py -m unifrac_weighted -t demo_input/GreenGenesCore-May09.ref.tre -e demo_input/fastunifrac_Ley_et_al_NRM_2_sample_id_map.txt -c demo_input/fastunifrac_Ley_et_al_NRM_2_sample_id_map-colors.txt
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
50 > scripts/scriptPcoa.py -m unifrac_unweighted -t demo_input/GreenGenesCore-May09.ref.tre -e demo_input/fastunifrac_Ley_et_al_NRM_2_sample_id_map.txt -c demo_input/fastunifrac_Ley_et_al_NRM_2_sample_id_map-colors.txt
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
51
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
52 There already exists a collection of functionality surrounding unifrac distances in Qiime and related software. We support these metrics here for completeness, if your need is not met here, please look into Qiime and related software for a solutions with a more rich collection of functionality.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
53
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
54 ## scriptManipulateTable.py ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
55 Abundance tables can be difficult to manipulate. This script captures frequent tasks that may be important to manipulating an abundance table including normalization, summing, filtering, stratifying the tables into subsets (for instance breaking up a large HMP table into tables, one for each body site), and other functionality. You will notice for every call you must give it the sample id (-i) and the last metadata which should be the row before your first data (-l). This helps the scripts understand what is a data measurement and what is a metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
56
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
57 _Remember you can do multiple tasks or use multiple arguments at the same time._
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
58 _Here is an example of summing, normalizing, adding on clade prefixes, and stratifies the tables based on the STSite metadata_
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
59
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
60 > scripts/scriptManipulateTable.py -i TID -l STSite -s -n -x -y STSite demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
61
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
62 Please look at the detailed description of normalization and summation for a clear understanding of how the data is being manipulated.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
63
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
64 *Manipulating the measurements*
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
65 A. How do I sum a table based on clade names?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
66
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
67 > scripts/scriptManipulateTable.py -i TID -l STSite -s demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
68
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
69 B. How do I normalize a table?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
70
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
71 > scripts/scriptManipulateTable.py -i TID -l STSite -n demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
72
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
73 *Filtering*
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
74 C. How do I filter a normalized table by percentage?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
75
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
76 This filters out bugs that are not in the top 0.95 percentage of at least 0.05 percent of the samples (a good default).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
77
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
78 > scripts/scriptManipulateTable.py -i TID -l STSite -P 0.95,0.05 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
79
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
80 D. How do I filter a normalized table by a minimum abundance?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
81
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
82 This filters out bugs that do not have at least a certain number of bugs in a certain number of samples. Here we show
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
83 the call to filter out all bugs which do not have at least 3 samples with at least 0.0001 abundance (a good initial default).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
84
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
85 > scripts/scriptManipulateTable.py -i TID -l STSite -A 0.0001,3 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
86
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
87 E. How do I filter a count table by count occurrence?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
88
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
89 This removes samples that do not have at least 5 counts in at least 3 samples (an initial default to use could be 2,2).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
90
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
91 > scripts/scriptManipulateTable.py -i TID -l STSite -O 5,3 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
92
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
93 F. How do I filter a table by standard deviation?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
94
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
95 > scripts/scriptManipulateTable.py -i TID -l STSite -D 1 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
96
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
97 *Filtering with knowledge of feature hierarchical relationship.*
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
98 F. How do I make the table have only terminal nodes?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
99
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
100 > scripts/scriptManipulateTable.py -i TID -l STSite -t demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
101
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
102 G. How do I remove all the OTUs from a table?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
103
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
104 > scripts/scriptManipulateTable.py -i TID -l STSite -u demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
105
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
106 H. How do I reduce all bugs more specific than a certain clade? Aka, how do I reset a table to be only a clade (genus) or higher?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
107
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
108 This reduces the bugs to bugs with 3 levels of hierarchy or less (class on a standard biological taxonomy).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
109
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
110 > scripts/scriptManipulateTable.py -i TID -l STSite -c 3 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
111
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
112 You may want to hierarchically sum all of you bugs before reducing the table to a certain level, just in case you are missing some.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
113
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
114 > scripts/scriptManipulateTable.py -i TID -l STSite -s -c 3 demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
115
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
116 Detail. OTUs or taxonomic clades are terminal nodes of a dendrogram representing the full taxonomy or phylogeny of a study. Biology may happen at these terminal clades or at higher level clades. Hierarchical summation uses the name of the bug (containing the consensus lineage) to add bugs together at different levels of their ancestral state and represent additional higher level clades or bigger groupings of bugs.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
117
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
118 More plainly, imagine if we have 2 bugs in a sample with 5 and 10 counts. These two bugs differ as species but share the rest of their ancestry. In this case, an additional bug is added for the genus level group.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
119
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
120 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1|s__species1 5
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
121 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1|s__species2 10
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
122 add
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
123 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1 15
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
124
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
125 A new kingdom, phylum, class, order, and family is not entered because they would be the same grouping of counts as the new genus level entry.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
126
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
127 If we had an additional bug
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
128 k__kingdom1|p__phylum2|c__class1|o__order2|f__family12|g__genus23|s__species14 2
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
129 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1|s__species1 5
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
130 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1|s__species2 10
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
131 add
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
132 k__kingdom1|p__phylum2|c__class1 17
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
133 k__kingdom1|p__phylum2|c__class1|o__order1|f__family1|g__genus1 15
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
134
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
135 Two new bugs are added because o__order1 and o__order2 can be combined at the c__class1 grouping and s__species1 and s__species2 can be combined at the g__genus1 level. Other groupings at other clade levels are not made because they represent the same groupings of counts already accounted for in the data by bugs and would be redundant. For instance, having a k__kingdom 17 count entry would be the same grouping as the c_class1 bug that was added and so is not created and added.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
136
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
137 *How do I reduce the table to a list of bugs?*
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
138
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
139 > scripts/scriptManipulateTable.py -i TID -l STSite -b features.txt demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
140 > scripts/scriptManipulateTable.py -i TID -l STSite -b 'Bacteria|3417,Bacteria|unclassified|4904' demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
141
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
142 IV. Manipulate samples by metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
143 J. How do I stratify the table to subtables based on a metadata? (Example. How do I take the HMP table and break it up by body site or time point?)
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
144
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
145 > scripts/scriptManipulateTable.py -i TID -l STSite -y STSite demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
146
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
147 K. How do I remove all samples of a certain metadata value? (Example, How do I remove all gut HMP body site samples but leave the rest in the table?)
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
148
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
149 > scripts/scriptManipulateTable.py -i TID -l STSite -r STSite,R_Retroauricular_crease, L_Retroauricular_crease demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
150
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
151 V. Manipulate the feature names
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
152 L. How do I add on the 'k__' and 's__' on the names of my bugs?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
153
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
154 > scripts/scriptManipulateTable.py -i TID -l STSite -x demo_input/Test.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
155
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
156 ## scriptPlotFeature.py ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
157 This script allows you to plot a row of an abundance table, metadata or data. This assumes the first column is the id and the remaining columns are values to be plotted. Three different plots can be generated based on the input arguments and the type of data given to the script. A boxplot is made if two features are given, one numeric and one categorical. A scatterplot is made if two numeric features are given. A histogram is made if one numeric feature is given.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
158
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
159 A. How do I plot a box plot of two data.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
160 A box plot requires two features, one not categorical and one that is categorical. The script detects this automatically and will plot the correct plot for you as you go.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
161
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
162 > scripts/scriptPlotFeature.py demo_input/Test.pcl STSite 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
163
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
164 B. How do I plot a scatter plot of two data?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
165 A box plot requires two features, both not categorical. The script detects this automatically and will plot the correct plot for you as you go.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
166
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
167 > scripts/scriptPlotFeature.py demo_input/Test.pcl 'Bacteria|unclassified|4904' 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
168
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
169 C. How do I plot a histogram of a feature?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
170 Just plot one numeric feature.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
171
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
172 > scripts/scriptPlotFeature.py demo_input/Test.pcl 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
173
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
174 D. How do I change the title or axes?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
175
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
176 > scripts/scriptPlotFeature.py -t Title -x Xaxis -y Yaxis demo_input/Test.pcl 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
177
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
178 E. How do I change the color? Use -c and a hex color.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
179
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
180 > scripts/scriptPlotFeature.py -c '#333333' demo_input/Test.pcl 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
181
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
182 F. How do I invert the colors for a black background? Use -r .
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
183
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
184 > scripts/scriptPlotFeature.py -r demo_input/Test.pcl 'Bacteria|3417'
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
185
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
186 ## scriptBiplotTSV.R ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
187 This script allows one to plot a tsv file as a biplot. A tsv file is a transposed PCL file (demo files are found in demo_input). The positioning of sample markers and bug text are generated by nonmetric multidimensional scaling. The metadata are represented by arrows and then a text at the head of the arrow. The coordinates of the arrows are determined by the center/average of the coordinates of the samples with that metadata showing a central tendency of where that metadata is located. More specifically, discontinuous metadata are broken down to levels (values), then each level is made into it's own binary metadata (0 for not having that value and 1 for having that value). Then for each new metadata, samples with the value of 1 are selected and have their coordinates in the ordination are averaged. This average coordinate set is then used as the coordinates for that metadata level. For continuous data, using the ordination coordinates for all the sample points, the value of the continuous metadata is placed in a landscape using the sample coordiantes as x and y and the z as the metadata value. This is then smoothed with a lowess and then the maximum fitted value's coordinates are used as the central tendency of the metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
188
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
189 This is a customizable plot there the metadata plotted, the bugs plotted, the arrow color, the bug and metadata text color, the sample colors, the sample shapes, and the title can be changed. Below are examples of how to use the commandline for this script; although options are shown seperately here, many options can be used together.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
190
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
191 A. This is the minimal call. The call to the script must include the following positional arguments lastmetadata and inputPCLFile after any optional arguments (given with flags preceeded by - or --).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
192
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
193 > ./scripts/scriptBiplotTSV.R STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
194
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
195 This consists of the script call, the last metadata value, and the input file.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
196
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
197 B. How do I specify an output file name? Use -o and then a name ending with the extension .pdf .
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
198
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
199 > ./scripts/scriptBiplotTSV.R -o Test2Biplot.pdf STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
200
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
201 C. How do I specify bug names to plot? Use -b and the names of the bugs to plot (as written in your pcl file) seperated by commas.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
202
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
203 > ./scripts/scriptBiplotTSV.R -b 'Bacteria|3417' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
204 > ./scripts/scriptBiplotTSV.R -b 'Bacteria|3417,Bacteria|unclassified|4904' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
205
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
206 D. How do I specify metadata to plot? Using the -m option, if plotting a continuous metadata, give the id of the metadata (first column entry of metadata), for any other metadata concatonate the metadata ID and the value of interest with "_" . Here are a working examples.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
207
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
208 > ./scripts/scriptBiplotTSV.R -m 'STSite_L_Antecubital_fossa,STSite_R_Antecubital_fossa' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
209 > ./scripts/scriptBiplotTSV.R -m 'Continuous' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
210
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
211 E. How do I specify a title? Use -i and the title text. Use ' to surround the text. This helps the command line understand that the text is together and not a series of commands. These should be used when you are giving a flag a value that has spaces or anything but alphanumeric characters.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
212
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
213 > ./scripts/scriptBiplotTSV.R -i 'Test Title' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
214
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
215 F. How do I specify a metadata to shape markers by? Use -y and the id for your metadata in your pcl file (the entry for your metadata in the first column).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
216
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
217 > ./scripts/scriptBiplotTSV.R -y STSite STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
218
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
219 G. How do I specify specific shapes to use? This requires the combination of -y and -s . Use -y to specify the metadata to use. Use -s to specify what shapes should be used for what metadata values. These are given as metadatavalue:shape,metadataValue:shape
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
220
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
221 > ./scripts/scriptBiplotTSV.R -y STSite -s 'L_Antecubital_fossa:15,R_Antecubital_fossa:23' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
222
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
223 H. How do I specify a metadata to color markers by? Use -c and the id for your metadata in your pcl file (the entry for your metadata in the first column).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
224
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
225 > ./scripts/scriptBiplotTSV.R -c STSite STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
226
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
227 I. How do I specify a default marker shape to use instead of using a metadata? Use -d and a number recognized by R's pch parameter (number between 1-25). For more information http://www.statmethods.net/advgraphs/parameters.html
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
228
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
229 > ./scripts/scriptBiplotTSV.R -d 1 STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
230
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
231 J. How do I specify a color range to use when coloring? Use -r with two (R supported) colors seperated by a comma. R supported colors can be found in many sources including this one http://www.stats4stem.org/r-colors.html
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
232
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
233 > ./scripts/scriptBiplotTSV.R -r 'red,cyan' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
234
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
235 K. How do I specify a color to use when drawing arrows? Use -a and a (R supported) color. R supported colors can be found in many sources including this one http://www.stats4stem.org/r-colors.html
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
236
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
237 > ./scripts/scriptBiplotTSV.R -a red STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
238
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
239 L. How do I specify a color to use for arrow text? Use -w and a (R supported) color. R supported colors can be found in many sources including this one http://www.stats4stem.org/r-colors.html
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
240
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
241 > ./scripts/scriptBiplotTSV.R -w orange STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
242
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
243 M. How do I specify a color to use for bug text? Use -t and a (R supported) color. Make sure to use -b to plot bugs. R supported colors can be found in many sources including this one http://www.stats4stem.org/r-colors.html
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
244
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
245 > ./scripts/scriptBiplotTSV.R -t pink -b 'Bacteria|3417' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
246
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
247 N. How do I rotate the projection (plot) in reference to a specific metadata? Use the -e option and give the plot a metadata and a weight for that metadata, the larger the weight, the more the rotation takes into account the metadata. You may have to experiement with different weights and see how the rotation is affected. The weights can be from 0 (no rotation by the metadata) to a very large number. The metadata name should be the metadata id if the value is continuous or the metadata id and the value (level) of interest seperated by a _ if the metadata is not continuous. Below are two examples, one using a continuous metadata and one using a discontinuous metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
248
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
249 > ./scripts/scriptBiplotTSV.R -e 'Continuous,2' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
250 > ./scripts/scriptBiplotTSV.R -e 'STSite_L_Antecubital_fossa,.5' STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
251
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
252 O. How do I color NAs a specific color, no matter other coloring in the plot? Use -n and a color supported by R. R supported colors can be found in many sources including this one http://www.stats4stem.org/r-colors.html This requires you to be coloring the plot by a metadata (option -c).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
253
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
254 > ./scripts/scriptBiplotTSV.R -n grey -c STSite STSite demo_input/Test-BiplotNA.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
255
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
256 P. How do I scale arrows in the plot. Use -z and a number to weight how much the metadata influences the rotation (number between 0 and very large).
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
257
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
258 > ./scripts/scriptBiplotTSV.R -z 2 STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
259
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
260 Q. How do I plot metadata labels without the arrows?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
261
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
262 > ./scripts/scriptBiplotTSV.R -A STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
263
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
264 R. How do I plot the biplot without metadata?
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
265
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
266 > ./scripts/scriptBiplotTSV.R -m "" STSite demo_input/Test-Biplot.tsv
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
267
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
268 ## scriptConvertBetweenBIOMAndPCL.py ##
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
269 The script allows one to convert between PCL and BIOM file formats. ID, last feature (row) metadata, and last sample metadata are optional information in the script call (when converting from PCL to BIOM). These are used to dictate placement of certain key sample metadata in the PCL file. Typically, it is helpful to set these arguments. This aids in the consistent and reliable manipulation of these files. If the are not given, a guess will be made to the ID and it will be assumed no metadata exist.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
270
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
271 A quick definition:
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
272 *ID or sample id* - typically your first row in the PCL file (the Ids of all your samples) in the example below "ID"
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
273 *Feature (row) metadata* - columns in your PCL file which describe your features. These come after your feature IDs but before your measurements.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
274 *Sample metadata* - rows in your PCL file which come before your measurements and describe your samples
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
275
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
276 For a description of a PCL and it's parts please look in the docs folder for PCL-Description.txt
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
277
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
278 A. The minimal call to convert from BIOM file to a PCL file or visa versa. This call indicates the sample metadata entry which is the sample id and which is the last listed metadata in a pcl file (before the data measurements). When converting a PCL file, if there are no metadata and only a metadata id, -l and -i is not required. If there are multiple metadata in a pcl file the -l (last metadata) field is required. Neither of these fields are required for biom file conversion to pcl.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
279
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
280 > ./scripts/scriptConvertBetweenBIOMAndPCL.py demo_input/Test_no_metadata.pcl example1.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
281 > ./scripts/scriptConvertBetweenBIOMAndPCL.py demo_input/Test.biom example2.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
282 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -l STSite demo_input/Test.pcl example3.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
283
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
284 B. Specifying ID and lastmetadata
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
285
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
286 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite demo_input/Test.pcl example4.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
287 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite demo_input/Test.biom example5.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
288
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
289 C. The case where there are no sample metadata, just sample IDs. Indicate the ID and if no last metadata is indicated (-l) it is assumed no sample metadata exist.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
290
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
291 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i ID demo_input/Test_no_metadata.pcl example6.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
292 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i ID demo_input/Test_no_metadata.biom example7.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
293
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
294 D. The case when converting a PCL file with Feature (row) metadata (for example taxonomy_5). Include the last column with feature metadata.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
295
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
296 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i ID -r taxonomy_5 -l STSite ./demo_input/testFeatureMetadata.pcl testFeatureMetadata.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
297
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
298 E. Although the output file name can be automatically generated, the output file name can be given if needed.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
299
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
300 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite demo_input/Test.biom CustomFileName.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
301 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite demo_input/Test.pcl CustomFileName.biom
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
302
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
303 F. Indicate the use of a pcl file using a delimiter that is not tab or indicate the creation of a pcl file using a delimier that is not tab.
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
304
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
305 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite -f , demo_input/Test-comma.pcl
2f4f6f08c8c4 Uploaded
george-weingart
parents:
diff changeset
306 > ./scripts/scriptConvertBetweenBIOMAndPCL.py -i TID -l STSite -f , demo_input/Test-comma.biom