默认 surefire 插件输出结果几种常见的格式：
AccountUpdatePrivacyCommentTest.setUpBeforeClass:69 1
AccountUpdatePrivacyCommentTest.testAttentionByComment:136
Expected: is "省略"
but: was "省略"
com.weibo.cases.maincase.XuelianCommentsHotTimelineWithFilterHotTimelineCommentBVTTest.testFilterCreate(com.weibo.cases.maincase.XuelianCommentsHotTimelineWithFilterHotTimelineCommentBVTTest)
Run 1: XuelianCommentsHotTimelineWithFilterHotTimelineCommentBVTTest.setUp:79 NullPointer
Run 2: XuelianCommentsHotTimelineWithFilterHotTimelineCommentBVTTest.tearDown:110
Expected: is "true"
but: was null
StatusShoppingTitleStatusTest.testFriendsTimelineRecom:245->createPage:319 This is error,should create page success!
如果断言是用 hamcrest，输出结果为多行，因为 hamcrest 源码使用换行符"\n""打印实际和预期：
https://github.com/hamcrest/JavaHamcrest
path:
JavaHamcrest/hamcrest-core/src/main/java/org/hamcrest/MatcherAssert.java
public static <T> void assertThat(String reason, T actual, Matcher<? super T> matcher) {
if (!matcher.matches(actual)) {
Description description = new StringDescription();
description.appendText(reason)
.appendText("\nExpected: ")
.appendDescriptionOf(matcher)
.appendText("\n but: ");
matcher.describeMismatch(actual, description);
throw new AssertionError(description.toString());
}
}
为了便于 mvn test 迭代重试，故要统一输出格式，否则需正则匹配各种不样的格式，开销大且易漏掉失败用例，修改了 maven surefire 源码，统一输出结果并将成功和 skipped 的用例信息也打印出来，方便统计结果信息：
package org.apache.maven.plugin.surefire.report;
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import org.apache.maven.plugin.surefire.StartupReportConfiguration;
import org.apache.maven.plugin.surefire.runorder.StatisticsReporter;
import org.apache.maven.surefire.report.DefaultDirectConsoleReporter;
import org.apache.maven.surefire.report.ReporterFactory;
import org.apache.maven.surefire.report.RunListener;
import org.apache.maven.surefire.report.RunStatistics;
import org.apache.maven.surefire.report.StackTraceWriter;
import org.apache.maven.surefire.suite.RunResult;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
/**
* Provides reporting modules on the plugin side.
* <p/>
* Keeps a centralized count of test run results.
*
* @author Kristian Rosenvold
*
* modify by hugang stdout success and skipped
*/
public class DefaultReporterFactory implements ReporterFactory {
private RunStatistics globalStats = new RunStatistics();
private final StartupReportConfiguration reportConfiguration;
private final StatisticsReporter statisticsReporter;
private final List<TestSetRunListener> listeners = Collections
.synchronizedList(new ArrayList<TestSetRunListener>());
// from "<testclass>.<testmethod>" -> statistics about all the runs for
// flaky tests
private Map<String, List<TestMethodStats>> flakyTests;
// from "<testclass>.<testmethod>" -> statistics about all the runs for
// failed tests
private Map<String, List<TestMethodStats>> failedTests;
// from "<testclass>.<testmethod>" -> statistics about all the runs for
// error tests
private Map<String, List<TestMethodStats>> errorTests;
// added by hugang, from "<testclass>.<testmethod>" -> statistics about all
// the runs for success tests
private Map<String, List<TestMethodStats>> successTests;
// added by hugang, from "<testclass>.<testmethod>" -> statistics about all
// the runs for skipped tests
private Map<String, List<TestMethodStats>> skippedTests;
public DefaultReporterFactory(StartupReportConfiguration reportConfiguration) {
this.reportConfiguration = reportConfiguration;
this.statisticsReporter = reportConfiguration
.instantiateStatisticsReporter();
}
public RunListener createReporter() {
return createTestSetRunListener();
}
public void mergeFromOtherFactories(
Collection<DefaultReporterFactory> factories) {
for (DefaultReporterFactory factory : factories) {
for (TestSetRunListener listener : factory.listeners) {
listeners.add(listener);
}
}
}
public RunListener createTestSetRunListener() {
TestSetRunListener testSetRunListener = new TestSetRunListener(
reportConfiguration.instantiateConsoleReporter(),
reportConfiguration.instantiateFileReporter(),
reportConfiguration.instantiateStatelessXmlReporter(),
reportConfiguration.instantiateConsoleOutputFileReporter(),
statisticsReporter, reportConfiguration.isTrimStackTrace(),
ConsoleReporter.PLAIN.equals(reportConfiguration
.getReportFormat()),
reportConfiguration.isBriefOrPlainFormat());
listeners.add(testSetRunListener);
return testSetRunListener;
}
public void addListener(TestSetRunListener listener) {
listeners.add(listener);
}
public RunResult close() {
mergeTestHistoryResult();
runCompleted();
for (TestSetRunListener listener : listeners) {
listener.close();
}
return globalStats.getRunResult();
}
private DefaultDirectConsoleReporter createConsoleLogger() {
return new DefaultDirectConsoleReporter(
reportConfiguration.getOriginalSystemOut());
}
// 测试开始
public void runStarting() {
final DefaultDirectConsoleReporter consoleReporter = createConsoleLogger();
consoleReporter.info("");
consoleReporter
.info("-------------------------------------------------------");
consoleReporter.info(" T E S T S");
consoleReporter
.info("-------------------------------------------------------");
}
// 测试结束
private void runCompleted() {
final DefaultDirectConsoleReporter logger = createConsoleLogger();
if (reportConfiguration.isPrintSummary()) {
logger.info("");
logger.info("Results :");
logger.info("");
}
// 输出不同类型用例信息
boolean printedFailures = printTestFailures(logger,
TestResultType.failure);
printedFailures |= printTestFailures(logger, TestResultType.error);
printedFailures |= printTestFailures(logger, TestResultType.flake);
// added by hugang， 添加success and skipped用例详细
boolean printedSuccessSkipped = printTestSuccessSkipped(logger,
TestResultType.success);
printedSuccessSkipped |= printTestSuccessSkipped(logger,
TestResultType.skipped);
if (printedFailures) {
logger.info("");
}
if (printedSuccessSkipped) {
logger.info("");
}
// 输出failed 和 error的用例集， 作为下次重跑物料
// private Map<String, List<TestMethodStats>> failedTests;
// private Map<String, List<TestMethodStats>> errorTests;
// logger.info( failedTests.toString() );
// logger.info( "" );
// logger.info( errorTests.toString() );
// globalStats.getSummary() 打印数量
logger.info(globalStats.getSummary());
logger.info("");
}
public RunStatistics getGlobalRunStatistics() {
mergeTestHistoryResult();
return globalStats;
}
public static DefaultReporterFactory defaultNoXml() {
return new DefaultReporterFactory(
StartupReportConfiguration.defaultNoXml());
}
/**
* Get the result of a test based on all its runs. If it has success and
* failures/errors, then it is a flake; if it only has errors or failures,
* then count its result based on its first run
*
* @param reportEntryList
* the list of test run report type for a given test
* @param rerunFailingTestsCount
* configured rerun count for failing tests
* @return the type of test result
*/
// Use default visibility for testing
static TestResultType getTestResultType(
List<ReportEntryType> reportEntryList, int rerunFailingTestsCount) {
if (reportEntryList == null || reportEntryList.isEmpty()) {
return TestResultType.unknown;
}
boolean seenSuccess = false, seenFailure = false, seenError = false;
for (ReportEntryType resultType : reportEntryList) {
if (resultType == ReportEntryType.SUCCESS) {
seenSuccess = true;
} else if (resultType == ReportEntryType.FAILURE) {
seenFailure = true;
} else if (resultType == ReportEntryType.ERROR) {
seenError = true;
}
}
if (seenFailure || seenError) {
if (seenSuccess && rerunFailingTestsCount > 0) {
return TestResultType.flake;
} else {
if (seenError) {
return TestResultType.error;
} else {
return TestResultType.failure;
}
}
} else if (seenSuccess) {
return TestResultType.success;
} else {
return TestResultType.skipped;
}
}
/**
* Merge all the TestMethodStats in each TestRunListeners and put results
* into flakyTests, failedTests and errorTests, indexed by test class and
* method name. Update globalStatistics based on the result of the merge.
*/
void mergeTestHistoryResult() {
globalStats = new RunStatistics();
flakyTests = new TreeMap<String, List<TestMethodStats>>();
failedTests = new TreeMap<String, List<TestMethodStats>>();
errorTests = new TreeMap<String, List<TestMethodStats>>();
// added by hugang, 存success 和 skpped 用例信息
successTests = new TreeMap<String, List<TestMethodStats>>();
skippedTests = new TreeMap<String, List<TestMethodStats>>();
// key:测试方法， value:结果
Map<String, List<TestMethodStats>> mergedTestHistoryResult = new HashMap<String, List<TestMethodStats>>();
// Merge all the stats for tests from listeners
for (TestSetRunListener listener : listeners) {
// 遍历每个listener
List<TestMethodStats> testMethodStats = listener
.getTestMethodStats();
// 遍历每个测试方法结果
for (TestMethodStats methodStats : testMethodStats) {
List<TestMethodStats> currentMethodStats = mergedTestHistoryResult
.get(methodStats.getTestClassMethodName());
// 查看mergedTestHistoryResult中是否已存在methodStats.getTestClassMethodName()，第一次添加
if (currentMethodStats == null) {
currentMethodStats = new ArrayList<TestMethodStats>();
currentMethodStats.add(methodStats);
// 添加到map中，key:方法， value:方法结果
mergedTestHistoryResult.put(
methodStats.getTestClassMethodName(),
currentMethodStats);
} else {
// 方法多个结果
currentMethodStats.add(methodStats);
}
}
}
// Update globalStatistics by iterating through mergedTestHistoryResult
int completedCount = 0, skipped = 0;
// 遍历所有的类，判断每一个类中的方法执行结果，放到对应的map中；
// TestMethodStats每个测试方法信息
for (Map.Entry<String, List<TestMethodStats>> entry : mergedTestHistoryResult
.entrySet()) {
List<TestMethodStats> testMethodStats = entry.getValue();
String testClassMethodName = entry.getKey();
completedCount++;
// 将每个测试方法的执行结果添加到resultTypeList中
List<ReportEntryType> resultTypeList = new ArrayList<ReportEntryType>();
for (TestMethodStats methodStats : testMethodStats) {
resultTypeList.add(methodStats.getResultType());
}
TestResultType resultType = getTestResultType(resultTypeList,
reportConfiguration.getRerunFailingTestsCount());
// 根据一个类的不同方法执行结果，放到对应的map中
switch (resultType) {
case success:
// If there are multiple successful runs of the same test, count
// all of them
int successCount = 0;
for (ReportEntryType type : resultTypeList) {
if (type == ReportEntryType.SUCCESS) {
successCount++;
}
}
completedCount += successCount - 1;
// added by hugang, 把success 用例信息，put 到map中
successTests.put(testClassMethodName, testMethodStats);
break;
case skipped:
// added by hugang, 把skipped 用例信息，put 到map中
skippedTests.put(testClassMethodName, testMethodStats);
skipped++;
break;
case flake:
flakyTests.put(testClassMethodName, testMethodStats);
break;
case failure:
failedTests.put(testClassMethodName, testMethodStats);
break;
case error:
errorTests.put(testClassMethodName, testMethodStats);
break;
default:
throw new IllegalStateException("Get unknown test result type");
}
}
globalStats.set(completedCount, errorTests.size(), failedTests.size(),
skipped, flakyTests.size());
}
/**
* Print failed tests and flaked tests. A test is considered as a failed
* test if it failed/got an error with all the runs. If a test passes in
* ever of the reruns, it will be count as a flaked test
*
* @param logger
* the logger used to log information
* @param type
* the type of results to be printed, could be error, failure or
* flake
* @return {@code true} if printed some lines
*/
private String statckInfo = "CustomResult Failed StackTrace";
// Use default visibility for testing
boolean printTestFailures(DefaultDirectConsoleReporter logger,
TestResultType type) {
boolean printed = false;
final Map<String, List<TestMethodStats>> testStats;
switch (type) {
case failure:
testStats = failedTests;
break;
case error:
testStats = errorTests;
break;
case flake:
testStats = flakyTests;
break;
default:
return printed;
}
if (!testStats.isEmpty()) {
// 被注释，添加到每行用例信息前，便于正则匹配
// logger.info( type.getLogPrefix() );
printed = true;
}
// 遍历Map:testStats, value: List<TestMethodStats>
// 元素TestMethodStats：Maintains per-thread test result state for a single
// test method.
for (Map.Entry<String, List<TestMethodStats>> entry : testStats
.entrySet()) {
printed = true;
List<TestMethodStats> testMethodStats = entry.getValue();
if (testMethodStats.size() == 1) {
// 被注释
// No rerun, follow the original output format
// logger.info( " " + testMethodStats.get( 0
// ).getStackTraceWriter().smartTrimmedStackTrace() );
// added by hugang , 每行用例信息前，便于正则匹配
// 打印失败信息
// 将错误追踪栈中换行符去掉(hamcrest匹配器错误信息输出多行)，只输出一行，便于正则匹配
String strFailStrace = testMethodStats.get(0)
.getStackTraceWriter().smartTrimmedStackTrace();
logger.info(statckInfo + "@"
+ strFailStrace.replaceAll("\n", ""));
// 只打印失败的类方法，供解析出用例信息
// getTestClassMethodName()返回格式：
// com.weibo.cases.hugang.SurefirePluginTest.test1(com.weibo.cases.hugang.SurefirePluginTest)
// 或com.weibo.cases.maincase.XiaoyuGroupStatusStatusBVTTest.com.weibo.cases.maincase.XiaoyuGroupStatusStatusBVTTest
logger.info(type.getLogPrefix() + "@"
+ testMethodStats.get(0).getTestClassMethodName());
} else {
// 一个方法有多个结果，比如@BeforeClass,@Before中失败; @BeforeClass失败，输出类似：
// com.weibo.cases.maincase.XiaoyuGroupStatusStatusBVTTest.
// com.weibo.cases.maincase.XiaoyuGroupStatusStatusBVTTest
// logger.info( statckInfo + "@" + entry.getKey() );
// 将多个结果整合到一行
// 存1个测试方法中多个运行结果信息
List<String> mutFailInfo = new ArrayList<String>();
for (int i = 0; i < testMethodStats.size(); i++) {
StackTraceWriter failureStackTrace = testMethodStats.get(i)
.getStackTraceWriter();
if (failureStackTrace == null) {
mutFailInfo.add(" Run " + (i + 1) + ": PASS");
} else {
mutFailInfo.add(" Run "
+ (i + 1)
+ ": "
+ failureStackTrace.smartTrimmedStackTrace()
.replaceAll("\n", ""));
}
}
String runInfo = "";
for (int j = 0; j < mutFailInfo.size(); j++) {
runInfo += mutFailInfo.get(j);
}
String allFailInfo = statckInfo + "@" + entry.getKey()
+ runInfo;
// 打印多个失败集, 统一放到一行，便于匹配
logger.info(allFailInfo);
// 打印失败的类方法， 供解析出用例信息
logger.info(type.getLogPrefix() + "@"
+ testMethodStats.get(0).getTestClassMethodName());
logger.info("");
}
}
return printed;
}
// 打印成功和skipped用例
boolean printTestSuccessSkipped(DefaultDirectConsoleReporter logger,
TestResultType type) {
boolean printed = false;
final Map<String, List<TestMethodStats>> testStats;
switch (type) {
// added by hugang；支持success and skipped
case success:
testStats = successTests;
break;
case skipped:
testStats = skippedTests;
break;
default:
return printed;
}
if (!testStats.isEmpty()) {
// 被注释，添加到每行用例信息前，便于正则匹配
// logger.info( type.getLogPrefix() );
printed = true;
}
for (Map.Entry<String, List<TestMethodStats>> entry : testStats
.entrySet()) {
printed = true;
List<TestMethodStats> testMethodStats = entry.getValue();
if (testMethodStats.size() == 1) {
// 被注释
// No rerun, follow the original output format
// logger.info( " " + testMethodStats.get( 0
// ).getStackTraceWriter().smartTrimmedStackTrace() );
// added by hugang , 每行用例信息前，便于正则匹配
logger.info(type.getLogPrefix() + "@"
+ testMethodStats.get(0).getTestClassMethodName());
} else {
logger.info(entry.getKey());
for (int i = 0; i < testMethodStats.size(); i++) {
logger.info(type.getLogPrefix() + "@"
+ testMethodStats.get(i).getTestClassMethodName());
}
logger.info("");
}
}
return printed;
}
// Describe the result of a given test
static enum TestResultType {
error("CustomResult Error"), failure("CustomResult Fail"), flake(
"CustomResult Flaked"), success("CustomResult Success"), skipped(
"CustomResult Skipped"), unknown("CustomResult Unknown");
private final String logPrefix;
private TestResultType(String logPrefix) {
this.logPrefix = logPrefix;
}
public String getLogPrefix() {
return logPrefix;
}
}
}
mvn test -Dtest=TestClass时，终端结果输出格式统一输出如下（并且每条用例信息都为 1 行，方便正则匹配）：CustomResult Failed StackTrace@错误栈信息
CustomResult Fail@失败信息
CustomResult Error@失败信息
CustomResult Skipped@skipped信息
CustomResult Success@成功信息
默认 surefire Result：
maven-surefire-customresult Result:
https://github.com/neven7/maven-surefire-customresult
选择分支 Branch: extensions-2.19
下载源码
版本已经修改为 2.19.1
1.本地使用，进入项目根目录，执行mvn clean -Dcheckstyle.skip=true -Dmaven.test.skip=true -DuniqueVersion=false -Denforcer.skip=true install 本地安装， 在测试项目 pom.xml 中引用如下：
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<forkMode>pertest</forkMode>
<argLine>-Xms1024m -Xmx1024m</argLine>
<printSummary>true</printSummary>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.19</version>
</dependency>
</dependencies>
</plugin>
</plugins>
2.公司内部使用，进入项目根目录，执行mvn clean -Dcheckstyle.skip=true -Dmaven.test.skip=true -DuniqueVersion=false -Denforcer.skip=true deploy 需要在项目 pom.xml 配置自己公司的私有仓库：
<distributionManagement>
<repository>
<id>weiboqa</id>
<name>weiboqacontent</name>
<url>http://ip:port/nexus/content/repositories/weiboqa</url>
</repository>
</distributionManagement>
在 settings.xml 配置私有仓库的账号和密码才能发布到私有仓库，发布成功后，测试项目 pom.xml 引用如上。
